2024-06-01T00:33:31.595 INFO:root:teuthology version: 0.0.1.dev301+g036ad57 2024-06-01T00:33:31.595 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi --verbose --archive /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360 --owner scheduled_teuthology@teuthology --description upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360/orig.config.yaml 2024-06-01T00:33:31.656 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-01T00:33:31.749 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360 branch: squid description: upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7735360' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: f1316ee19712baed461a5bda1a1cefe32b1d255a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8008 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 0/32 suite: upgrade suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f1316ee19712baed461a5bda1a1cefe32b1d255a targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAGNqoLn8wVT+9NUVZL2f8wq8PjFd8MJOGc9ldI3/p/znKYJ2+ffVCo/G8U0YHcNYoVbJ/nSN6B3cg69uk91fAc= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEorxXyhNLWbP17vs6o16Ap6ODNMmAEaiGcYJS6xOSJgV20dRW6bDsWD9a73+55EWFo+tb/B9ksNwqO+BP/zLo= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 036ad575e36eb2f5d36bafd4f61b4922eadba901 timestamp: 2024-05-31_02:08:10 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-01T00:33:31.749 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa; will attempt to use it 2024-06-01T00:33:31.750 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks 2024-06-01T00:33:31.750 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-01T00:33:31.751 INFO:teuthology.task.internal:Checking packages... 2024-06-01T00:33:31.772 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-01T00:33:31.772 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T00:33:31.773 INFO:teuthology.packaging:ref: None 2024-06-01T00:33:31.773 INFO:teuthology.packaging:tag: None 2024-06-01T00:33:31.773 INFO:teuthology.packaging:branch: squid 2024-06-01T00:33:31.773 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:33:31.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-01T00:33:31.987 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2946.g3d89accf 2024-06-01T00:33:31.988 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-01T00:33:31.995 INFO:teuthology.task.internal:no buildpackages task found 2024-06-01T00:33:31.995 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-01T00:33:32.012 INFO:teuthology.task.internal:Saving configuration 2024-06-01T00:33:32.039 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-01T00:33:32.047 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-01T00:33:32.073 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-01 00:28:14.655568', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAGNqoLn8wVT+9NUVZL2f8wq8PjFd8MJOGc9ldI3/p/znKYJ2+ffVCo/G8U0YHcNYoVbJ/nSN6B3cg69uk91fAc='} 2024-06-01T00:33:32.099 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-01 00:28:14.657181', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEorxXyhNLWbP17vs6o16Ap6ODNMmAEaiGcYJS6xOSJgV20dRW6bDsWD9a73+55EWFo+tb/B9ksNwqO+BP/zLo='} 2024-06-01T00:33:32.099 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-01T00:33:32.105 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-01T00:33:32.105 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-01T00:33:32.105 INFO:teuthology.run_tasks:Running task console_log... 2024-06-01T00:33:32.175 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f59ed70bac0>, signals=[15]) 2024-06-01T00:33:32.175 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-01T00:33:32.182 INFO:teuthology.task.internal:Opening connections... 2024-06-01T00:33:32.182 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-06-01T00:33:32.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:33:32.259 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-06-01T00:33:32.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:33:32.332 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-01T00:33:32.337 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-06-01T00:33:32.355 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-06-01T00:33:32.356 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-01T00:33:32.413 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-06-01T00:33:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-06-01T00:33:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-01T00:33:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-06-01T00:33:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-01T00:33:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-01T00:33:32.414 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-01T00:33:32.414 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-06-01T00:33:32.492 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-06-01T00:33:32.510 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-06-01T00:33:32.510 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-06-01T00:33:32.567 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-01T00:33:32.568 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-01T00:33:32.569 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-06-01T00:33:32.592 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-01T00:33:32.600 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-01T00:33:32.607 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-01T00:33:32.608 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-06-01T00:33:32.611 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-06-01T00:33:32.627 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-01T00:33:32.636 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-01T00:33:32.636 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-06-01T00:33:32.668 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-06-01T00:33:32.788 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-01T00:33:32.848 INFO:teuthology.run_tasks:Running task kernel... 2024-06-01T00:33:32.864 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-01T00:33:32.864 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-01T00:33:32.864 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-01T00:33:32.865 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-01T00:33:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:33:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:33:32.882 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-06-01T00:33:32.882 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-06-01T00:33:32.938 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-437.el9.x86_64 2024-06-01T00:33:32.939 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.14.0-437.el9.x86_64 2024-06-01T00:33:32.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2024-06-01T00:33:32.940 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-437.el9.x86_64 2024-06-01T00:33:32.940 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-437.el9.x86_64 2024-06-01T00:33:32.940 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-06-01T00:33:35.138 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.1 MB 00:00 2024-06-01T00:33:39.565 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.1 MB 00:04 2024-06-01T00:33:39.880 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 19 MB 00:03 2024-06-01T00:33:44.189 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.2 MB 00:00 2024-06-01T00:33:46.039 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 17 kB 00:00 2024-06-01T00:33:47.510 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-06-01T00:33:52.707 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-06-01T00:33:54.697 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-01T00:33:54.697 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-01T00:33:54.798 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:33:54.808 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:33:54.808 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-06-01T00:33:54.808 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:33:54.808 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:33:54.808 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-06-01T00:33:54.809 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:33:54.810 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 80 M 2024-06-01T00:33:54.810 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 124 M 2024-06-01T00:33:54.810 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:33:55.158 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 4.3 MB/s | 893 kB 00:00 2024-06-01T00:33:56.017 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-06-01T00:33:56.497 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-06-01T00:33:58.633 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 8.3 MB/s | 29 MB 00:03 2024-06-01T00:33:58.975 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 8.6 MB/s | 35 MB 00:04 2024-06-01T00:33:58.975 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:33:58.975 INFO:teuthology.orchestra.run.smithi079.stdout:Total 19 MB/s | 80 MB 00:04 2024-06-01T00:33:59.675 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:34:00.026 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:34:00.026 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:34:00.974 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:34:00.975 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:34:02.747 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.2 MB 00:02 2024-06-01T00:34:03.389 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:34:04.130 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-01T00:34:04.225 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T00:34:04.981 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 17 kB 00:01 2024-06-01T00:34:06.799 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-06-01T00:34:06.876 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T00:34:07.026 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T00:34:11.792 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T00:34:12.048 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:34:12.149 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 5.3 kB/s | 1.7 kB 00:00 2024-06-01T00:34:14.256 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-01T00:34:14.256 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-01T00:34:14.332 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:34:14.341 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:34:14.341 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-06-01T00:34:14.342 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:34:14.343 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 80 M 2024-06-01T00:34:14.343 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 124 M 2024-06-01T00:34:14.343 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:34:15.409 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 957 kB/s | 893 kB 00:00 2024-06-01T00:34:16.782 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:34:25.341 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): kernel-modules-core-5.14.0-452.el9.x86_6 2.9 MB/s | 29 MB 00:09 2024-06-01T00:34:29.282 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): kernel-core-5.14.0-452.el9.x86_64.rpm 1.1 MB/s | 16 MB 00:14 2024-06-01T00:34:46.147 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:34:46.186 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:34:46.507 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:34:46.507 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-01T00:34:46.507 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T00:34:46.507 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T00:34:47.346 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:34:47.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:34:47.347 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:34:47.347 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-01T00:34:47.347 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-01T00:34:47.347 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-01T00:34:47.347 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-01T00:34:47.347 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:34:47.347 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:34:47.721 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T00:34:47.839 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T00:34:47.840 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T00:34:47.840 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-01T00:34:47.840 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-01T00:34:47.840 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-01T00:34:47.840 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-01T00:34:47.840 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-06-01T00:34:47.840 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T00:34:47.892 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T00:34:47.892 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T00:34:47.892 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-06-01T00:34:47.923 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-06-01T00:34:47.923 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-06-01T00:34:47.923 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-06-01T00:34:47.923 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 68.el9 2024-06-01T00:34:47.923 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8264254 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-01T00:34:47.924 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-01T00:34:47.925 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-06-01T00:34:47.926 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:34:47.926 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-06-01T00:34:47.927 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-01T00:34:47.927 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-452.el9.x86_64 2024-06-01T00:34:47.927 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-01T00:34:48.521 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-06-01T00:34:49.910 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-01T00:34:49.937 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-06-01T00:34:49.939 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-06-01T00:34:49.959 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.7YkKy362Cj 2024-06-01T00:34:49.960 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7YkKy362Cj 2024-06-01T00:34:50.024 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.7YkKy362Cj 2024-06-01T00:34:50.170 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.7YkKy362Cj is 6KB 2024-06-01T00:34:50.218 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.7YkKy362Cj 2024-06-01T00:34:50.232 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-06-01T00:34:50.302 INFO:teuthology.orchestra.run.smithi079.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-01T00:34:50.302 INFO:teuthology.orchestra.run.smithi079.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-01T00:34:50.302 INFO:teuthology.orchestra.run.smithi079.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-01T00:34:50.303 INFO:teuthology.orchestra.run.smithi079.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-01T00:34:50.303 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-01T00:34:50.446 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-06-01T00:34:56.713 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 838 kB/s | 35 MB 00:42 2024-06-01T00:34:56.713 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:34:56.713 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.9 MB/s | 80 MB 00:42 2024-06-01T00:34:57.407 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:34:57.786 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:34:57.786 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:34:58.750 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:34:58.751 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:35:01.144 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:35:01.833 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-01T00:35:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T00:35:04.571 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T00:35:04.715 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T00:35:09.493 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T00:35:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:35:14.415 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:35:20.449 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-01T00:35:20.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-06-01T00:35:20.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:35:38.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-06-01T00:35:43.745 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:35:43.784 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:35:44.185 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:35:44.185 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-01T00:35:44.186 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T00:35:44.186 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T00:35:44.991 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T00:35:44.991 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:35:44.991 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:35:44.991 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-01T00:35:44.992 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-01T00:35:44.992 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-01T00:35:44.992 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-01T00:35:44.992 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:35:44.992 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:35:45.359 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T00:35:45.438 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T00:35:45.439 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T00:35:45.439 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-01T00:35:45.439 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-01T00:35:45.439 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-01T00:35:45.439 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-01T00:35:45.439 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2024-06-01T00:35:45.440 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T00:35:45.492 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T00:35:45.493 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T00:35:45.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2024-06-01T00:35:45.529 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2024-06-01T00:35:45.529 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.06 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 68.el9 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Group : Unspecified 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 8264254 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-01T00:35:45.530 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : builder@centos.org 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:35:45.531 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2024-06-01T00:35:45.532 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-01T00:35:45.532 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 5.14.0-452.el9.x86_64 2024-06-01T00:35:45.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-01T00:35:46.141 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2024-06-01T00:35:47.550 INFO:teuthology.orchestra.run.smithi160.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-01T00:35:47.595 INFO:teuthology.orchestra.run.smithi160.stderr:done 2024-06-01T00:35:47.596 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2024-06-01T00:35:47.612 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.UknkarqepJ 2024-06-01T00:35:47.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UknkarqepJ 2024-06-01T00:35:47.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.UknkarqepJ 2024-06-01T00:35:47.811 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.UknkarqepJ is 6KB 2024-06-01T00:35:47.860 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.UknkarqepJ 2024-06-01T00:35:47.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2024-06-01T00:35:47.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-06-01T00:35:47.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:35:47.936 INFO:teuthology.orchestra.run.smithi160.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-01T00:35:47.936 INFO:teuthology.orchestra.run.smithi160.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-01T00:35:47.936 INFO:teuthology.orchestra.run.smithi160.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-01T00:35:47.936 INFO:teuthology.orchestra.run.smithi160.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-01T00:35:47.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-01T00:35:48.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2024-06-01T00:35:50.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-06-01T00:36:02.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-06-01T00:36:02.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:36:18.104 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-01T00:36:18.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-06-01T00:36:18.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:36:34.844 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-06-01T00:36:35.279 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-06-01T00:36:35.279 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-452.el9.x86_64"... 2024-06-01T00:36:35.279 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-06-01T00:36:35.293 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-452.el9.x86_64 2024-06-01T00:36:35.294 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-01T00:36:35.294 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-01T00:36:35.294 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-01T00:36:36.295 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-01T00:36:36.295 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-01T00:36:36.399 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-06-01T00:36:36.434 DEBUG:teuthology.parallel:result is None 2024-06-01T00:36:36.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-06-01T00:36:45.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-06-01T00:36:45.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:36:48.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-06-01T00:37:00.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-06-01T00:37:00.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:37:19.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-06-01T00:37:34.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-06-01T00:37:34.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:37:37.051 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-06-01T00:37:37.549 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-06-01T00:37:37.549 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-452.el9.x86_64"... 2024-06-01T00:37:37.549 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-06-01T00:37:37.564 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-452.el9.x86_64 2024-06-01T00:37:37.564 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-01T00:37:37.564 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-01T00:37:37.564 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-01T00:37:38.565 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-01T00:37:38.565 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-01T00:37:38.657 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2024-06-01T00:37:38.703 DEBUG:teuthology.parallel:result is None 2024-06-01T00:37:38.704 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-01T00:37:38.711 INFO:teuthology.task.internal:Creating test directory... 2024-06-01T00:37:38.711 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-01T00:37:38.714 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-01T00:37:38.731 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-01T00:37:38.742 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-01T00:37:38.755 INFO:teuthology.task.internal:Creating archive directory... 2024-06-01T00:37:38.755 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-01T00:37:38.773 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-01T00:37:38.839 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-01T00:37:38.850 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-01T00:37:38.851 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-01T00:37:38.853 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-01T00:37:38.887 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T00:37:38.894 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T00:37:38.897 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T00:37:38.905 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T00:37:38.907 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-01T00:37:38.915 INFO:teuthology.task.internal:Configuring sudo... 2024-06-01T00:37:38.916 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-01T00:37:38.949 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-01T00:37:38.976 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-01T00:37:38.989 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-01T00:37:38.989 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-01T00:37:39.018 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-01T00:37:39.033 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T00:37:39.121 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T00:37:39.166 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T00:37:39.250 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T00:37:39.289 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:37:39.289 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-01T00:37:39.355 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T00:37:39.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T00:37:39.443 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T00:37:39.524 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T00:37:39.566 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:37:39.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-01T00:37:39.633 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-06-01T00:37:39.635 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-06-01T00:37:39.680 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T00:37:39.711 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T00:37:40.143 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-01T00:37:40.151 INFO:teuthology.task.internal:Starting timer... 2024-06-01T00:37:40.151 INFO:teuthology.run_tasks:Running task pcp... 2024-06-01T00:37:40.165 INFO:teuthology.run_tasks:Running task selinux... 2024-06-01T00:37:40.178 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-01T00:37:40.179 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-06-01T00:37:40.257 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-06-01T00:37:40.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2024-06-01T00:37:40.321 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2024-06-01T00:37:40.323 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-01T00:37:40.323 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-06-01T00:37:40.356 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-06-01T00:37:40.356 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2024-06-01T00:37:40.379 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2024-06-01T00:37:40.379 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2024-06-01T00:37:40.380 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-01T00:37:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:37:40.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-01T00:37:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:37:40.451 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-01T00:37:40.451 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-06-01T00:37:40.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2024-06-01T00:37:40.527 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-01T00:37:40.540 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-01T00:37:40.543 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-01T00:37:40.668 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-01T00:37:40.679 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-01T00:37:40.680 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-01T00:42:12.338 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2024-06-01T00:42:12.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-06-01T00:42:12.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:42:12.419 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-06-01T00:42:12.511 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-06-01T00:42:12.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-06-01T00:42:12.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T00:42:12.606 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-06-01T00:42:12.687 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-06-01T00:42:12.688 INFO:teuthology.run_tasks:Running task clock... 2024-06-01T00:42:12.699 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-01T00:42:12.699 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-01T00:42:12.700 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T00:42:12.702 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-01T00:42:12.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T00:42:12.729 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-01T00:42:12.744 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-01T00:42:12.760 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-01T00:42:12.767 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-06-01T00:42:12.776 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-01T00:42:12.778 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-06-01T00:42:12.792 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-01T00:42:12.801 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2024-06-01T00:42:12.806 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-01T00:42:12.812 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2024-06-01T00:42:12.827 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-01T00:42:12.841 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-01T00:42:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-06-01T00:42:12.865 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T00:42:12.865 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-06-01T00:42:12.865 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.865 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.865 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.865 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.912 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-06-01T00:42:12.915 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T00:42:12.915 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-06-01T00:42:12.915 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.915 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.915 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.916 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:42:12.916 INFO:teuthology.run_tasks:Running task install... 2024-06-01T00:42:12.924 DEBUG:teuthology.task.install:project ceph 2024-06-01T00:42:12.925 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-01T00:42:12.925 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-01T00:42:12.925 INFO:teuthology.task.install:Using flavor: default 2024-06-01T00:42:12.932 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-01T00:42:12.932 INFO:teuthology.task.install:extra packages: [] 2024-06-01T00:42:12.932 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-01T00:42:12.933 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T00:42:12.933 INFO:teuthology.packaging:ref: None 2024-06-01T00:42:12.933 INFO:teuthology.packaging:tag: None 2024-06-01T00:42:12.933 INFO:teuthology.packaging:branch: reef 2024-06-01T00:42:12.933 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:42:12.933 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-01T00:42:12.936 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-01T00:42:12.936 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T00:42:12.936 INFO:teuthology.packaging:ref: None 2024-06-01T00:42:12.936 INFO:teuthology.packaging:tag: None 2024-06-01T00:42:12.936 INFO:teuthology.packaging:branch: reef 2024-06-01T00:42:12.936 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:42:12.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-01T00:42:13.255 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/ 2024-06-01T00:42:13.256 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1387.g71c5cfc9 2024-06-01T00:42:13.299 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/ 2024-06-01T00:42:13.299 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1387.g71c5cfc9 2024-06-01T00:42:13.414 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-01T00:42:13.414 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:42:13.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-01T00:42:13.441 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-01T00:42:13.442 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:42:13.442 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-01T00:42:13.443 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-01T00:42:13.443 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T00:42:13.443 INFO:teuthology.packaging:ref: None 2024-06-01T00:42:13.443 INFO:teuthology.packaging:tag: None 2024-06-01T00:42:13.443 INFO:teuthology.packaging:branch: reef 2024-06-01T00:42:13.444 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:42:13.444 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-01T00:42:13.471 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-01T00:42:13.471 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T00:42:13.471 INFO:teuthology.packaging:ref: None 2024-06-01T00:42:13.471 INFO:teuthology.packaging:tag: None 2024-06-01T00:42:13.471 INFO:teuthology.packaging:branch: reef 2024-06-01T00:42:13.471 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:42:13.472 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-01T00:42:13.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-01T00:42:13.539 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-01T00:42:13.584 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-01T00:42:13.615 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-01T00:42:13.648 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2024-06-01T00:42:13.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2024-06-01T00:42:13.679 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-06-01T00:42:13.680 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-06-01T00:42:13.894 INFO:teuthology.orchestra.run.smithi160.stdout:45 files removed 2024-06-01T00:42:13.909 INFO:teuthology.orchestra.run.smithi079.stdout:45 files removed 2024-06-01T00:42:13.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2024-06-01T00:42:13.936 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-06-01T00:42:14.599 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 193 kB/s | 77 kB 00:00 2024-06-01T00:42:14.681 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 164 kB/s | 77 kB 00:00 2024-06-01T00:42:14.925 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 41 kB/s | 11 kB 00:00 2024-06-01T00:42:15.040 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-06-01T00:42:15.284 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-06-01T00:42:15.391 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-06-01T00:42:15.890 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.1 MB 00:00 2024-06-01T00:42:15.971 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-06-01T00:42:17.677 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-06-01T00:42:21.769 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-06-01T00:42:24.236 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 17 kB 00:01 2024-06-01T00:42:25.140 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-06-01T00:42:26.507 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-06-01T00:42:29.433 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.2 MB 00:00 2024-06-01T00:42:31.042 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 17 kB 00:00 2024-06-01T00:42:31.663 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-06-01T00:42:32.643 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-06-01T00:42:33.979 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:42:33.989 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:42:33.989 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:42:33.989 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:42:33.989 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:42:33.989 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 7.7 M 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 3.3 M 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 3.0 M 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 5.1 M 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 18 M 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 25 k 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-01T00:42:33.990 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 709 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 475 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 4.5 M 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 45 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 129 k 2024-06-01T00:42:33.991 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 162 k 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 320 k 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 298 k 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 99 k 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout:Install 28 Packages 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2024-06-01T00:42:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:42:33.993 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 53 M 2024-06-01T00:42:33.993 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:42:35.372 INFO:teuthology.orchestra.run.smithi160.stdout:(1/30): ceph-radosgw-18.2.2-1387.g71c5cfc9.el9. 7.5 MB/s | 7.7 MB 00:01 2024-06-01T00:42:35.431 INFO:teuthology.orchestra.run.smithi160.stdout:(2/30): ceph-selinux-18.2.2-1387.g71c5cfc9.el9. 432 kB/s | 25 kB 00:00 2024-06-01T00:42:35.556 INFO:teuthology.orchestra.run.smithi160.stdout:(3/30): libcephfs2-18.2.2-1387.g71c5cfc9.el9.x8 5.5 MB/s | 709 kB 00:00 2024-06-01T00:42:35.632 INFO:teuthology.orchestra.run.smithi160.stdout:(4/30): libradosstriper1-18.2.2-1387.g71c5cfc9. 6.2 MB/s | 475 kB 00:00 2024-06-01T00:42:35.823 INFO:teuthology.orchestra.run.smithi160.stdout:(5/30): ceph-base-18.2.2-1387.g71c5cfc9.el9.x86 3.4 MB/s | 5.1 MB 00:01 2024-06-01T00:42:35.915 INFO:teuthology.orchestra.run.smithi160.stdout:(6/30): python3-ceph-argparse-18.2.2-1387.g71c5 491 kB/s | 45 kB 00:00 2024-06-01T00:42:36.149 INFO:teuthology.orchestra.run.smithi160.stdout:(7/30): ceph-common-18.2.2-1387.g71c5cfc9.el9.x 10 MB/s | 18 MB 00:01 2024-06-01T00:42:36.166 INFO:teuthology.orchestra.run.smithi160.stdout:(8/30): python3-ceph-common-18.2.2-1387.g71c5cf 516 kB/s | 129 kB 00:00 2024-06-01T00:42:36.224 INFO:teuthology.orchestra.run.smithi160.stdout:(9/30): librgw2-18.2.2-1387.g71c5cfc9.el9.x86_6 7.5 MB/s | 4.5 MB 00:00 2024-06-01T00:42:36.250 INFO:teuthology.orchestra.run.smithi160.stdout:(10/30): python3-cephfs-18.2.2-1387.g71c5cfc9.e 1.6 MB/s | 162 kB 00:00 2024-06-01T00:42:36.275 INFO:teuthology.orchestra.run.smithi160.stdout:(11/30): python3-rados-18.2.2-1387.g71c5cfc9.el 2.9 MB/s | 320 kB 00:00 2024-06-01T00:42:36.300 INFO:teuthology.orchestra.run.smithi160.stdout:(12/30): python3-rgw-18.2.2-1387.g71c5cfc9.el9. 1.9 MB/s | 99 kB 00:00 2024-06-01T00:42:36.325 INFO:teuthology.orchestra.run.smithi160.stdout:(13/30): python3-rbd-18.2.2-1387.g71c5cfc9.el9. 2.9 MB/s | 298 kB 00:00 2024-06-01T00:42:36.426 INFO:teuthology.orchestra.run.smithi160.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-06-01T00:42:36.451 INFO:teuthology.orchestra.run.smithi160.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 609 kB/s | 107 kB 00:00 2024-06-01T00:42:36.477 INFO:teuthology.orchestra.run.smithi160.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 911 kB/s | 160 kB 00:00 2024-06-01T00:42:36.518 INFO:teuthology.orchestra.run.smithi160.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-06-01T00:42:36.544 INFO:teuthology.orchestra.run.smithi160.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 633 kB/s | 42 kB 00:00 2024-06-01T00:42:36.569 INFO:teuthology.orchestra.run.smithi160.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-06-01T00:42:37.144 INFO:teuthology.orchestra.run.smithi160.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 177 kB/s | 111 kB 00:00 2024-06-01T00:42:37.262 INFO:teuthology.orchestra.run.smithi160.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-06-01T00:42:37.412 INFO:teuthology.orchestra.run.smithi160.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 356 kB/s | 309 kB 00:00 2024-06-01T00:42:37.445 INFO:teuthology.orchestra.run.smithi160.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 264 kB/s | 49 kB 00:00 2024-06-01T00:42:37.595 INFO:teuthology.orchestra.run.smithi160.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-06-01T00:42:37.793 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-06-01T00:42:37.862 INFO:teuthology.orchestra.run.smithi160.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 719 kB/s | 192 kB 00:00 2024-06-01T00:42:38.529 INFO:teuthology.orchestra.run.smithi160.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 774 kB/s | 839 kB 00:01 2024-06-01T00:42:39.088 INFO:teuthology.orchestra.run.smithi160.stdout:(27/30): librados2-18.2.2-1387.g71c5cfc9.el9.x8 5.9 MB/s | 3.3 MB 00:00 2024-06-01T00:42:39.405 INFO:teuthology.orchestra.run.smithi160.stdout:(28/30): librbd1-18.2.2-1387.g71c5cfc9.el9.x86_ 9.5 MB/s | 3.0 MB 00:00 2024-06-01T00:42:39.830 INFO:teuthology.orchestra.run.smithi160.stdout:(29/30): thrift-0.15.0-2.el9.x86_64.rpm 823 kB/s | 1.6 MB 00:01 2024-06-01T00:42:40.159 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:42:40.162 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:42:40.162 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:42:40.162 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 7.7 M 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 3.3 M 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 3.0 M 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 5.1 M 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 18 M 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 25 k 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-06-01T00:42:40.163 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 709 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 475 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 4.5 M 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-01T00:42:40.164 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 45 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 129 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 162 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 320 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 298 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 99 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:42:40.165 INFO:teuthology.orchestra.run.smithi079.stdout:Install 28 Packages 2024-06-01T00:42:40.166 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-06-01T00:42:40.166 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:42:40.166 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 53 M 2024-06-01T00:42:40.166 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:42:41.260 INFO:teuthology.orchestra.run.smithi079.stdout:(1/30): ceph-radosgw-18.2.2-1387.g71c5cfc9.el9. 8.1 MB/s | 7.7 MB 00:00 2024-06-01T00:42:41.352 INFO:teuthology.orchestra.run.smithi079.stdout:(2/30): ceph-base-18.2.2-1387.g71c5cfc9.el9.x86 4.9 MB/s | 5.1 MB 00:01 2024-06-01T00:42:41.369 INFO:teuthology.orchestra.run.smithi079.stdout:(3/30): ceph-selinux-18.2.2-1387.g71c5cfc9.el9. 232 kB/s | 25 kB 00:00 2024-06-01T00:42:41.461 INFO:teuthology.orchestra.run.smithi079.stdout:(4/30): libcephfs2-18.2.2-1387.g71c5cfc9.el9.x8 6.4 MB/s | 709 kB 00:00 2024-06-01T00:42:41.494 INFO:teuthology.orchestra.run.smithi079.stdout:(5/30): libradosstriper1-18.2.2-1387.g71c5cfc9. 3.7 MB/s | 475 kB 00:00 2024-06-01T00:42:41.553 INFO:teuthology.orchestra.run.smithi079.stdout:(6/30): python3-ceph-argparse-18.2.2-1387.g71c5 770 kB/s | 45 kB 00:00 2024-06-01T00:42:41.620 INFO:teuthology.orchestra.run.smithi079.stdout:(7/30): python3-ceph-common-18.2.2-1387.g71c5cf 1.9 MB/s | 129 kB 00:00 2024-06-01T00:42:41.687 INFO:teuthology.orchestra.run.smithi079.stdout:(8/30): python3-cephfs-18.2.2-1387.g71c5cfc9.el 2.4 MB/s | 162 kB 00:00 2024-06-01T00:42:41.804 INFO:teuthology.orchestra.run.smithi079.stdout:(9/30): python3-rados-18.2.2-1387.g71c5cfc9.el9 2.7 MB/s | 320 kB 00:00 2024-06-01T00:42:42.063 INFO:teuthology.orchestra.run.smithi079.stdout:(10/30): ceph-common-18.2.2-1387.g71c5cfc9.el9. 10 MB/s | 18 MB 00:01 2024-06-01T00:42:42.088 INFO:teuthology.orchestra.run.smithi079.stdout:(11/30): python3-rbd-18.2.2-1387.g71c5cfc9.el9. 1.0 MB/s | 298 kB 00:00 2024-06-01T00:42:42.113 INFO:teuthology.orchestra.run.smithi079.stdout:(12/30): python3-rgw-18.2.2-1387.g71c5cfc9.el9. 1.9 MB/s | 99 kB 00:00 2024-06-01T00:42:42.180 INFO:teuthology.orchestra.run.smithi079.stdout:(13/30): librgw2-18.2.2-1387.g71c5cfc9.el9.x86_ 6.2 MB/s | 4.5 MB 00:00 2024-06-01T00:42:42.449 INFO:teuthology.orchestra.run.smithi160.stdout:(30/30): libarrow-9.0.0-11.el9.x86_64.rpm 769 kB/s | 4.4 MB 00:05 2024-06-01T00:42:42.451 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:42:42.451 INFO:teuthology.orchestra.run.smithi160.stdout:Total 6.3 MB/s | 53 MB 00:08 2024-06-01T00:42:42.522 INFO:teuthology.orchestra.run.smithi079.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-06-01T00:42:42.525 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:42:42.547 INFO:teuthology.orchestra.run.smithi079.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 233 kB/s | 107 kB 00:00 2024-06-01T00:42:42.562 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:42:42.562 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:42:42.614 INFO:teuthology.orchestra.run.smithi079.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 320 kB/s | 160 kB 00:00 2024-06-01T00:42:42.715 INFO:teuthology.orchestra.run.smithi079.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 422 kB/s | 42 kB 00:00 2024-06-01T00:42:42.815 INFO:teuthology.orchestra.run.smithi079.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2024-06-01T00:42:42.965 INFO:teuthology.orchestra.run.smithi079.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-06-01T00:42:43.083 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:42:43.084 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:42:43.157 INFO:teuthology.orchestra.run.smithi079.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 250 kB/s | 111 kB 00:00 2024-06-01T00:42:43.266 INFO:teuthology.orchestra.run.smithi079.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-06-01T00:42:43.366 INFO:teuthology.orchestra.run.smithi079.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 561 kB/s | 309 kB 00:00 2024-06-01T00:42:43.391 INFO:teuthology.orchestra.run.smithi079.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-06-01T00:42:43.475 INFO:teuthology.orchestra.run.smithi079.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-06-01T00:42:43.617 INFO:teuthology.orchestra.run.smithi079.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-06-01T00:42:43.725 INFO:teuthology.orchestra.run.smithi079.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.5 MB/s | 839 kB 00:00 2024-06-01T00:42:43.867 INFO:teuthology.orchestra.run.smithi079.stdout:(27/30): libarrow-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2024-06-01T00:42:43.943 INFO:teuthology.orchestra.run.smithi079.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-06-01T00:42:44.134 INFO:teuthology.orchestra.run.smithi079.stdout:(29/30): librados2-18.2.2-1387.g71c5cfc9.el9.x8 8.1 MB/s | 3.3 MB 00:00 2024-06-01T00:42:44.193 INFO:teuthology.orchestra.run.smithi079.stdout:(30/30): librbd1-18.2.2-1387.g71c5cfc9.el9.x86_ 9.2 MB/s | 3.0 MB 00:00 2024-06-01T00:42:44.195 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:42:44.195 INFO:teuthology.orchestra.run.smithi079.stdout:Total 13 MB/s | 53 MB 00:04 2024-06-01T00:42:44.269 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:42:44.305 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:42:44.305 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:42:44.395 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:42:44.498 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-06-01T00:42:44.526 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-06-01T00:42:44.585 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-06-01T00:42:44.819 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:42:44.820 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:42:44.965 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-06-01T00:42:45.017 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/32 2024-06-01T00:42:45.142 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/32 2024-06-01T00:42:45.225 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 6/32 2024-06-01T00:42:45.294 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 7/32 2024-06-01T00:42:45.356 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 7/32 2024-06-01T00:42:45.384 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-06-01T00:42:45.429 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:18.2.2-1387.g71c5cfc9.el9. 9/32 2024-06-01T00:42:45.493 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 10/32 2024-06-01T00:42:45.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_6 11/32 2024-06-01T00:42:45.595 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_6 11/32 2024-06-01T00:42:45.700 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-06-01T00:42:45.736 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-06-01T00:42:45.778 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-06-01T00:42:45.809 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-06-01T00:42:45.828 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-06-01T00:42:45.864 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-06-01T00:42:45.897 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-06-01T00:42:46.161 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:42:46.196 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-06-01T00:42:46.243 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-06-01T00:42:46.243 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 20/32 2024-06-01T00:42:46.273 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-06-01T00:42:46.331 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-06-01T00:42:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 20/32 2024-06-01T00:42:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 21/32 2024-06-01T00:42:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-06-01T00:42:46.701 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-06-01T00:42:46.747 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/32 2024-06-01T00:42:46.802 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-06-01T00:42:46.822 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/32 2024-06-01T00:42:46.829 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 24/32 2024-06-01T00:42:46.903 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 24/32 2024-06-01T00:42:46.907 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 6/32 2024-06-01T00:42:46.948 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 7/32 2024-06-01T00:42:46.977 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 25/32 2024-06-01T00:42:47.007 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x8 26/32 2024-06-01T00:42:47.011 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 7/32 2024-06-01T00:42:47.039 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-06-01T00:42:47.075 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:18.2.2-1387.g71c5cfc9.el9. 9/32 2024-06-01T00:42:47.141 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 10/32 2024-06-01T00:42:47.167 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_6 11/32 2024-06-01T00:42:47.233 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_6 11/32 2024-06-01T00:42:47.339 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-06-01T00:42:47.375 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-06-01T00:42:47.416 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-06-01T00:42:47.448 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-06-01T00:42:47.474 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-06-01T00:42:47.511 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-06-01T00:42:47.543 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-06-01T00:42:47.844 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-06-01T00:42:47.889 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 20/32 2024-06-01T00:42:47.980 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 20/32 2024-06-01T00:42:48.011 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 21/32 2024-06-01T00:42:48.047 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-06-01T00:42:48.448 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-06-01T00:42:48.475 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 24/32 2024-06-01T00:42:48.552 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 24/32 2024-06-01T00:42:48.624 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 25/32 2024-06-01T00:42:48.653 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x8 26/32 2024-06-01T00:42:49.075 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 27/32 2024-06-01T00:42:49.123 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 27/32 2024-06-01T00:42:49.679 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 27/32 2024-06-01T00:42:49.727 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 28/32 2024-06-01T00:42:49.826 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 28/32 2024-06-01T00:42:49.826 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-01T00:42:49.826 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-01T00:42:49.826 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:42:49.853 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/32 2024-06-01T00:42:50.753 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 27/32 2024-06-01T00:42:50.785 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 27/32 2024-06-01T00:42:51.316 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 27/32 2024-06-01T00:42:51.373 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 28/32 2024-06-01T00:42:51.471 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 28/32 2024-06-01T00:42:51.471 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-01T00:42:51.471 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-01T00:42:51.471 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:42:51.499 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/32 2024-06-01T00:42:58.934 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/32 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /root 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2024-06-01T00:42:58.935 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:42:59.672 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/32 2024-06-01T00:42:59.697 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/32 2024-06-01T00:42:59.697 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:42:59.697 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-01T00:42:59.697 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-01T00:42:59.697 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-01T00:42:59.698 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:42:59.699 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-01T00:42:59.782 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-01T00:42:59.783 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-01T00:43:00.855 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-01T00:43:00.856 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/32 2024-06-01T00:43:00.856 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/32 2024-06-01T00:43:00.856 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/32 2024-06-01T00:43:00.856 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/32 2024-06-01T00:43:00.856 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_6 6/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 7/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1387.g71c5cfc9.el9. 8/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x8 9/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 10/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 11/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 12/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 13/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-06-01T00:43:00.857 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-06-01T00:43:00.858 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-06-01T00:43:00.859 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-06-01T00:43:00.859 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-06-01T00:43:00.859 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-06-01T00:43:00.859 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/32 2024-06-01T00:43:00.859 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-06-01T00:43:00.859 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 31/32 2024-06-01T00:43:01.015 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/32 2024-06-01T00:43:01.015 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-06-01T00:43:01.015 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-06-01T00:43:01.016 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-06-01T00:43:01.016 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-06-01T00:43:01.016 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-06-01T00:43:01.016 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-06-01T00:43:01.016 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-06-01T00:43:01.016 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-01T00:43:01.633 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: fmt-8.1.1-5.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.635 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.636 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:01.636 INFO:teuthology.orchestra.run.smithi160.stdout: re2-1:20211101-3.el9.x86_64 2024-06-01T00:43:01.636 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-01T00:43:01.636 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:01.636 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:01.742 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/32 2024-06-01T00:43:01.768 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/32 2024-06-01T00:43:01.768 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:01.768 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-01T00:43:01.768 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-01T00:43:01.769 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-01T00:43:01.769 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:01.770 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-01T00:43:01.878 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-01T00:43:01.878 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-01T00:43:02.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2024-06-01T00:43:02.585 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:31 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:02.706 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:02.706 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:02.706 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:43:02.706 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:02.706 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:43:02.706 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 40 M 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 166 k 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-06-01T00:43:02.707 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:02.708 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 40 M 2024-06-01T00:43:02.708 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 163 M 2024-06-01T00:43:02.708 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:43:03.012 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 305 kB 00:00 2024-06-01T00:43:03.025 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_6 6/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 7/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1387.g71c5cfc9.el9. 8/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x8 9/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 10/32 2024-06-01T00:43:03.026 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 11/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 12/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 13/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-06-01T00:43:03.027 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-06-01T00:43:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-06-01T00:43:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-06-01T00:43:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-06-01T00:43:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-06-01T00:43:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/32 2024-06-01T00:43:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-06-01T00:43:03.028 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 31/32 2024-06-01T00:43:03.062 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-06-01T00:43:03.079 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): libcephsqlite-18.2.2-1387.g71c5cfc9.el9. 674 kB/s | 166 kB 00:00 2024-06-01T00:43:03.813 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.814 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: fmt-8.1.1-5.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-01T00:43:03.815 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:03.816 INFO:teuthology.orchestra.run.smithi079.stdout: re2-1:20211101-3.el9.x86_64 2024-06-01T00:43:03.817 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-01T00:43:03.817 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:03.817 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:04.160 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-06-01T00:43:04.743 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:27 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:04.864 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:04.864 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 40 M 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 166 k 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:04.865 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-06-01T00:43:04.866 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:04.866 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 40 M 2024-06-01T00:43:04.866 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 163 M 2024-06-01T00:43:04.866 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:05.099 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 305 kB 00:00 2024-06-01T00:43:05.262 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): ceph-test-18.2.2-1387.g71c5cfc9.el9.x86_ 16 MB/s | 40 MB 00:02 2024-06-01T00:43:05.264 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:05.265 INFO:teuthology.orchestra.run.smithi160.stdout:Total 16 MB/s | 40 MB 00:02 2024-06-01T00:43:05.274 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): libcephsqlite-18.2.2-1387.g71c5cfc9.el9. 448 kB/s | 166 kB 00:00 2024-06-01T00:43:05.291 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:43:05.311 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:43:05.312 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:43:05.626 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:43:05.627 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:43:05.982 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 72 kB/s | 64 kB 00:00 2024-06-01T00:43:06.095 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:43:06.186 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-01T00:43:06.237 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-01T00:43:06.264 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/4 2024-06-01T00:43:06.832 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): ceph-test-18.2.2-1387.g71c5cfc9.el9.x86_ 21 MB/s | 40 MB 00:01 2024-06-01T00:43:06.834 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:06.835 INFO:teuthology.orchestra.run.smithi079.stdout:Total 20 MB/s | 40 MB 00:01 2024-06-01T00:43:06.862 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:06.882 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:06.882 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:07.200 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:07.201 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:07.669 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:07.750 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-01T00:43:07.806 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-01T00:43:07.834 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/4 2024-06-01T00:43:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/4 2024-06-01T00:43:10.080 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/4 2024-06-01T00:43:11.513 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/4 2024-06-01T00:43:11.592 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/4 2024-06-01T00:43:12.067 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/4 2024-06-01T00:43:12.067 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/4 2024-06-01T00:43:12.067 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/4 2024-06-01T00:43:12.067 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-01T00:43:12.349 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-01T00:43:12.349 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:12.350 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:43:12.350 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:12.350 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:12.350 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-01T00:43:12.350 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-01T00:43:12.350 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:12.350 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:12.595 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2024-06-01T00:43:13.178 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:42 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:13.302 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:13.304 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:13.304 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:43:13.304 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:13.304 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 6.5 k 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 2.1 M 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 1.5 M 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 246 k 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 4.7 M 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 17 M 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-01T00:43:13.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-01T00:43:13.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:13.307 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:43:13.308 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:13.308 INFO:teuthology.orchestra.run.smithi160.stdout:Install 33 Packages 2024-06-01T00:43:13.308 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:13.308 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 29 M 2024-06-01T00:43:13.308 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 104 M 2024-06-01T00:43:13.308 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:43:13.711 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/4 2024-06-01T00:43:13.711 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/4 2024-06-01T00:43:13.711 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/4 2024-06-01T00:43:13.711 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-01T00:43:13.977 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-01T00:43:13.977 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:13.977 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:13.977 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:13.978 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:13.978 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-01T00:43:13.978 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-01T00:43:13.978 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:13.978 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:14.105 INFO:teuthology.orchestra.run.smithi160.stdout:(1/33): ceph-18.2.2-1387.g71c5cfc9.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2024-06-01T00:43:14.246 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-06-01T00:43:14.347 INFO:teuthology.orchestra.run.smithi160.stdout:(2/33): ceph-mgr-18.2.2-1387.g71c5cfc9.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-06-01T00:43:14.824 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:37 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:14.947 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 6.5 k 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 2.1 M 2024-06-01T00:43:14.949 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 1.5 M 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 246 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 4.7 M 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 17 M 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-01T00:43:14.950 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-01T00:43:14.951 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout:Install 33 Packages 2024-06-01T00:43:14.952 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:14.953 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 29 M 2024-06-01T00:43:14.953 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 104 M 2024-06-01T00:43:14.953 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:14.980 INFO:teuthology.orchestra.run.smithi160.stdout:(3/33): ceph-mon-18.2.2-1387.g71c5cfc9.el9.x86_ 5.3 MB/s | 4.7 MB 00:00 2024-06-01T00:43:15.048 INFO:teuthology.orchestra.run.smithi160.stdout:(4/33): ceph-mds-18.2.2-1387.g71c5cfc9.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-06-01T00:43:15.073 INFO:teuthology.orchestra.run.smithi160.stdout:(5/33): ceph-mgr-modules-core-18.2.2-1387.g71c5 2.6 MB/s | 246 kB 00:00 2024-06-01T00:43:15.290 INFO:teuthology.orchestra.run.smithi160.stdout:(6/33): ceph-osd-18.2.2-1387.g71c5cfc9.el9.x86_ 18 MB/s | 17 MB 00:00 2024-06-01T00:43:15.692 INFO:teuthology.orchestra.run.smithi079.stdout:(1/33): ceph-18.2.2-1387.g71c5cfc9.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-06-01T00:43:15.706 INFO:teuthology.orchestra.run.smithi160.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 256 kB/s | 106 kB 00:00 2024-06-01T00:43:15.740 INFO:teuthology.orchestra.run.smithi160.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 366 kB/s | 253 kB 00:00 2024-06-01T00:43:15.850 INFO:teuthology.orchestra.run.smithi160.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-06-01T00:43:15.891 INFO:teuthology.orchestra.run.smithi160.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 735 kB/s | 135 kB 00:00 2024-06-01T00:43:15.958 INFO:teuthology.orchestra.run.smithi160.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-06-01T00:43:16.001 INFO:teuthology.orchestra.run.smithi160.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-06-01T00:43:16.042 INFO:teuthology.orchestra.run.smithi160.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-06-01T00:43:16.059 INFO:teuthology.orchestra.run.smithi160.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 345 kB/s | 35 kB 00:00 2024-06-01T00:43:16.092 INFO:teuthology.orchestra.run.smithi160.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 553 kB/s | 51 kB 00:00 2024-06-01T00:43:16.108 INFO:teuthology.orchestra.run.smithi079.stdout:(2/33): ceph-mon-18.2.2-1387.g71c5cfc9.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-06-01T00:43:16.134 INFO:teuthology.orchestra.run.smithi160.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-06-01T00:43:16.167 INFO:teuthology.orchestra.run.smithi079.stdout:(3/33): ceph-mds-18.2.2-1387.g71c5cfc9.el9.x86_ 3.3 MB/s | 2.1 MB 00:00 2024-06-01T00:43:16.268 INFO:teuthology.orchestra.run.smithi079.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1387.g71c5 2.4 MB/s | 246 kB 00:00 2024-06-01T00:43:16.401 INFO:teuthology.orchestra.run.smithi079.stdout:(5/33): ceph-mgr-18.2.2-1387.g71c5cfc9.el9.x86_ 1.7 MB/s | 1.5 MB 00:00 2024-06-01T00:43:16.584 INFO:teuthology.orchestra.run.smithi160.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 83 kB/s | 43 kB 00:00 2024-06-01T00:43:16.685 INFO:teuthology.orchestra.run.smithi160.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-06-01T00:43:16.785 INFO:teuthology.orchestra.run.smithi160.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 248 kB/s | 172 kB 00:00 2024-06-01T00:43:16.802 INFO:teuthology.orchestra.run.smithi160.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-06-01T00:43:16.877 INFO:teuthology.orchestra.run.smithi160.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-06-01T00:43:16.893 INFO:teuthology.orchestra.run.smithi079.stdout:(6/33): ceph-osd-18.2.2-1387.g71c5cfc9.el9.x86_ 21 MB/s | 17 MB 00:00 2024-06-01T00:43:16.894 INFO:teuthology.orchestra.run.smithi160.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-06-01T00:43:16.918 INFO:teuthology.orchestra.run.smithi079.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 389 kB/s | 253 kB 00:00 2024-06-01T00:43:16.969 INFO:teuthology.orchestra.run.smithi160.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-06-01T00:43:17.003 INFO:teuthology.orchestra.run.smithi160.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-06-01T00:43:17.027 INFO:teuthology.orchestra.run.smithi079.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-06-01T00:43:17.078 INFO:teuthology.orchestra.run.smithi160.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 380 kB/s | 358 kB 00:00 2024-06-01T00:43:17.135 INFO:teuthology.orchestra.run.smithi079.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 1.1 MB/s | 125 kB 00:00 2024-06-01T00:43:17.153 INFO:teuthology.orchestra.run.smithi160.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 429 kB/s | 79 kB 00:00 2024-06-01T00:43:17.187 INFO:teuthology.orchestra.run.smithi160.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-06-01T00:43:17.252 INFO:teuthology.orchestra.run.smithi079.stdout:(10/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-06-01T00:43:17.287 INFO:teuthology.orchestra.run.smithi079.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 1.4 MB/s | 1.2 MB 00:00 2024-06-01T00:43:17.288 INFO:teuthology.orchestra.run.smithi160.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 358 kB/s | 36 kB 00:00 2024-06-01T00:43:17.338 INFO:teuthology.orchestra.run.smithi160.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 489 kB/s | 90 kB 00:00 2024-06-01T00:43:17.370 INFO:teuthology.orchestra.run.smithi079.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 417 kB/s | 35 kB 00:00 2024-06-01T00:43:17.395 INFO:teuthology.orchestra.run.smithi079.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 212 kB/s | 106 kB 00:00 2024-06-01T00:43:17.420 INFO:teuthology.orchestra.run.smithi079.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-06-01T00:43:17.445 INFO:teuthology.orchestra.run.smithi079.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-06-01T00:43:17.446 INFO:teuthology.orchestra.run.smithi160.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 615 kB/s | 272 kB 00:00 2024-06-01T00:43:17.521 INFO:teuthology.orchestra.run.smithi079.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 332 kB/s | 42 kB 00:00 2024-06-01T00:43:17.546 INFO:teuthology.orchestra.run.smithi160.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-06-01T00:43:17.638 INFO:teuthology.orchestra.run.smithi160.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 655 kB/s | 230 kB 00:00 2024-06-01T00:43:17.771 INFO:teuthology.orchestra.run.smithi079.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-06-01T00:43:17.880 INFO:teuthology.orchestra.run.smithi079.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-06-01T00:43:17.913 INFO:teuthology.orchestra.run.smithi160.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 742 kB/s | 427 kB 00:00 2024-06-01T00:43:17.918 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:17.918 INFO:teuthology.orchestra.run.smithi160.stdout:Total 6.3 MB/s | 29 MB 00:04 2024-06-01T00:43:17.930 INFO:teuthology.orchestra.run.smithi079.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 355 kB/s | 172 kB 00:00 2024-06-01T00:43:17.988 INFO:teuthology.orchestra.run.smithi079.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-06-01T00:43:18.030 INFO:teuthology.orchestra.run.smithi079.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-06-01T00:43:18.038 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:43:18.076 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:43:18.077 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:43:18.097 INFO:teuthology.orchestra.run.smithi079.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 622 kB/s | 358 kB 00:00 2024-06-01T00:43:18.123 INFO:teuthology.orchestra.run.smithi079.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-06-01T00:43:18.148 INFO:teuthology.orchestra.run.smithi079.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-06-01T00:43:18.231 INFO:teuthology.orchestra.run.smithi079.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-06-01T00:43:18.257 INFO:teuthology.orchestra.run.smithi079.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 291 kB/s | 46 kB 00:00 2024-06-01T00:43:18.332 INFO:teuthology.orchestra.run.smithi079.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-06-01T00:43:18.357 INFO:teuthology.orchestra.run.smithi079.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2024-06-01T00:43:18.382 INFO:teuthology.orchestra.run.smithi079.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2024-06-01T00:43:18.466 INFO:teuthology.orchestra.run.smithi079.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 267 kB/s | 36 kB 00:00 2024-06-01T00:43:18.472 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:43:18.472 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:43:18.508 INFO:teuthology.orchestra.run.smithi079.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-06-01T00:43:18.558 INFO:teuthology.orchestra.run.smithi079.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-06-01T00:43:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-06-01T00:43:18.585 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:18.585 INFO:teuthology.orchestra.run.smithi079.stdout:Total 8.0 MB/s | 29 MB 00:03 2024-06-01T00:43:18.708 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:18.745 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:18.745 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:19.122 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:19.123 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:19.614 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:43:19.682 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-01T00:43:19.723 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-01T00:43:19.756 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-01T00:43:19.849 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-01T00:43:19.918 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-01T00:43:19.968 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-01T00:43:20.002 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-01T00:43:20.032 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-01T00:43:20.067 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-01T00:43:20.108 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-01T00:43:20.149 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-01T00:43:20.184 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-01T00:43:20.218 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-01T00:43:20.282 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-01T00:43:20.294 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:20.357 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-01T00:43:20.362 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-01T00:43:20.393 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-01T00:43:20.426 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-01T00:43:20.496 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-01T00:43:20.518 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-01T00:43:20.575 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-01T00:43:20.605 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-01T00:43:20.636 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-01T00:43:20.645 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-01T00:43:20.674 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-01T00:43:20.680 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-01T00:43:20.710 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-01T00:43:20.723 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-01T00:43:20.745 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-01T00:43:20.779 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-01T00:43:20.788 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-01T00:43:20.819 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-01T00:43:20.854 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-01T00:43:20.888 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-01T00:43:20.926 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-01T00:43:20.979 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-01T00:43:21.000 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-01T00:43:21.028 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-01T00:43:21.107 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-01T00:43:21.146 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-01T00:43:21.220 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-01T00:43:21.226 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-01T00:43:21.260 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-01T00:43:21.279 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-01T00:43:21.326 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-01T00:43:21.375 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-01T00:43:21.377 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-01T00:43:21.439 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-01T00:43:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1387.g71c5cfc9.el9. 28/33 2024-06-01T00:43:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/33 2024-06-01T00:43:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/33 2024-06-01T00:43:21.580 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:21.580 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-01T00:43:21.580 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-01T00:43:21.580 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-01T00:43:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:21.631 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-01T00:43:21.680 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-01T00:43:21.750 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-01T00:43:21.838 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-01T00:43:21.879 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-01T00:43:21.995 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-01T00:43:22.145 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1387.g71c5cfc9.el9. 28/33 2024-06-01T00:43:22.173 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/33 2024-06-01T00:43:22.201 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 29/33 2024-06-01T00:43:22.202 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:22.202 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-01T00:43:22.202 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-01T00:43:22.202 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-01T00:43:22.202 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:23.056 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/33 2024-06-01T00:43:23.085 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/33 2024-06-01T00:43:23.086 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:23.086 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-01T00:43:23.086 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-01T00:43:23.086 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-01T00:43:23.086 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:23.531 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 31/33 2024-06-01T00:43:23.555 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 31/33 2024-06-01T00:43:23.555 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:23.556 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-01T00:43:23.556 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-01T00:43:23.556 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-01T00:43:23.556 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:23.678 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/33 2024-06-01T00:43:23.707 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 30/33 2024-06-01T00:43:23.707 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:23.707 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-01T00:43:23.707 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-01T00:43:23.707 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-01T00:43:23.707 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:23.757 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 32/33 2024-06-01T00:43:23.780 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 32/33 2024-06-01T00:43:23.780 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:23.780 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-01T00:43:23.780 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-01T00:43:23.781 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-01T00:43:23.781 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 33/33 2024-06-01T00:43:24.146 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 31/33 2024-06-01T00:43:24.169 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 31/33 2024-06-01T00:43:24.169 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:24.169 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-01T00:43:24.169 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-01T00:43:24.169 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-01T00:43:24.170 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:24.371 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 32/33 2024-06-01T00:43:24.394 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 32/33 2024-06-01T00:43:24.394 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:24.395 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-01T00:43:24.395 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-01T00:43:24.395 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-01T00:43:24.395 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:24.458 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 33/33 2024-06-01T00:43:25.580 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 33/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1387.g71c5cfc9.el9. 6/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-01T00:43:25.581 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-01T00:43:25.582 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-01T00:43:25.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-01T00:43:25.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-01T00:43:25.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-01T00:43:25.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-01T00:43:25.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-01T00:43:25.584 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-01T00:43:26.037 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 33/33 2024-06-01T00:43:26.037 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/33 2024-06-01T00:43:26.037 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/33 2024-06-01T00:43:26.037 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 3/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 4/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 5/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1387.g71c5cfc9.el9. 6/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-01T00:43:26.038 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-01T00:43:26.039 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-01T00:43:26.040 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-01T00:43:26.040 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-01T00:43:26.040 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-01T00:43:26.040 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-01T00:43:26.040 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.410 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-01T00:43:26.411 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply-3.11-14.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-01T00:43:26.412 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-01T00:43:26.413 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-01T00:43:26.413 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:26.413 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:26.787 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2024-06-01T00:43:26.871 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-01T00:43:26.871 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:26.871 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:26.871 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.871 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.871 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-01T00:43:26.872 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-01T00:43:26.873 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-01T00:43:26.874 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-01T00:43:26.874 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-01T00:43:26.874 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-01T00:43:26.874 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-01T00:43:26.874 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:26.874 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:27.235 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-06-01T00:43:27.375 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:56 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:27.456 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:43:27.499 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:27.500 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:43:27.500 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:27.562 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2024-06-01T00:43:27.824 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:50 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:27.906 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:43:27.948 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:27.949 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:43:27.949 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:28.024 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-06-01T00:43:28.138 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:57 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:28.263 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:28.264 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:28.264 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:43:28.264 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:28.264 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 224 k 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 224 k 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 216 k 2024-06-01T00:43:28.265 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:43:28.603 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:51 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:28.615 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-18.2.2-1387.g71c5cfc9.el9.noarch.rpm 640 kB/s | 224 kB 00:00 2024-06-01T00:43:28.616 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:28.616 INFO:teuthology.orchestra.run.smithi160.stdout:Total 637 kB/s | 224 kB 00:00 2024-06-01T00:43:28.616 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:43:28.622 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:43:28.622 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:43:28.624 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:43:28.625 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:43:28.695 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:43:28.730 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 224 k 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 224 k 2024-06-01T00:43:28.731 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 216 k 2024-06-01T00:43:28.732 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:28.978 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-18.2.2-1387.g71c5cfc9.el9.noarch.rpm 909 kB/s | 224 kB 00:00 2024-06-01T00:43:28.979 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:28.979 INFO:teuthology.orchestra.run.smithi079.stdout:Total 904 kB/s | 224 kB 00:00 2024-06-01T00:43:28.979 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:28.984 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:28.984 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:28.987 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:28.987 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:29.053 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:29.111 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:29.202 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:29.491 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:29.543 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:29.582 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:29.784 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:29.784 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:29.784 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:43:29.784 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:29.784 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:29.784 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:29.921 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:29.995 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2024-06-01T00:43:30.153 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/1 2024-06-01T00:43:30.154 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:30.154 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:30.154 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:30.154 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:30.154 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:30.410 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-06-01T00:43:30.579 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:59 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:30.704 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:30.704 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 142 k 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 142 k 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 395 k 2024-06-01T00:43:30.705 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:43:30.937 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-18.2.2-1387.g71c5cf 614 kB/s | 142 kB 00:00 2024-06-01T00:43:30.939 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:30.939 INFO:teuthology.orchestra.run.smithi160.stdout:Total 611 kB/s | 142 kB 00:00 2024-06-01T00:43:30.939 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:43:30.946 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:43:30.946 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:43:30.994 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:53 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:31.003 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:43:31.004 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:43:31.116 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:31.117 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 142 k 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-06-01T00:43:31.118 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:31.119 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 142 k 2024-06-01T00:43:31.119 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 395 k 2024-06-01T00:43:31.119 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:31.140 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:43:31.230 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9. 1/1 2024-06-01T00:43:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9. 1/1 2024-06-01T00:43:31.255 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:31.255 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-01T00:43:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:31.351 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-18.2.2-1387.g71c5cf 612 kB/s | 142 kB 00:00 2024-06-01T00:43:31.351 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:31.352 INFO:teuthology.orchestra.run.smithi079.stdout:Total 608 kB/s | 142 kB 00:00 2024-06-01T00:43:31.352 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:31.359 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:31.359 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:31.416 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:31.417 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:31.565 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:31.644 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9. 1/1 2024-06-01T00:43:31.668 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9. 1/1 2024-06-01T00:43:31.668 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:43:31.668 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-01T00:43:31.668 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:32.166 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9. 1/1 2024-06-01T00:43:32.166 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:32.167 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:43:32.167 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:32.167 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:32.167 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:32.369 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2024-06-01T00:43:32.541 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9. 1/1 2024-06-01T00:43:32.541 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:32.541 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:32.542 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:43:32.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:32.542 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:32.746 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-06-01T00:43:32.970 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:01 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:33.057 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:43:33.100 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:33.102 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:43:33.102 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:33.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2024-06-01T00:43:33.342 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:56 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:33.429 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:43:33.472 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:33.473 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:43:33.473 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:33.552 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-06-01T00:43:33.733 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:02 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:33.857 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 3.5 M 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 24 k 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 16 k 2024-06-01T00:43:33.859 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout:Install 5 Packages 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.8 M 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 87 M 2024-06-01T00:43:33.860 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:43:34.123 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:57 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:34.124 INFO:teuthology.orchestra.run.smithi160.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1387.g71c 130 kB/s | 24 kB 00:00 2024-06-01T00:43:34.224 INFO:teuthology.orchestra.run.smithi160.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1387.g71c5 57 kB/s | 16 kB 00:00 2024-06-01T00:43:34.250 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 3.5 M 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 24 k 2024-06-01T00:43:34.251 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 16 k 2024-06-01T00:43:34.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-01T00:43:34.252 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-01T00:43:34.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:34.252 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:34.252 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:34.252 INFO:teuthology.orchestra.run.smithi079.stdout:Install 5 Packages 2024-06-01T00:43:34.252 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:34.253 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.8 M 2024-06-01T00:43:34.253 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 87 M 2024-06-01T00:43:34.253 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:34.502 INFO:teuthology.orchestra.run.smithi160.stdout:(3/5): ceph-mgr-dashboard-18.2.2-1387.g71c5cfc9 6.3 MB/s | 3.5 MB 00:00 2024-06-01T00:43:34.638 INFO:teuthology.orchestra.run.smithi079.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1387.g71c 102 kB/s | 24 kB 00:00 2024-06-01T00:43:34.671 INFO:teuthology.orchestra.run.smithi079.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1387.g71c5 60 kB/s | 16 kB 00:00 2024-06-01T00:43:34.716 INFO:teuthology.orchestra.run.smithi160.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 52 kB/s | 31 kB 00:00 2024-06-01T00:43:34.822 INFO:teuthology.orchestra.run.smithi079.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 167 kB/s | 31 kB 00:00 2024-06-01T00:43:34.897 INFO:teuthology.orchestra.run.smithi079.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 838 kB/s | 188 kB 00:00 2024-06-01T00:43:35.025 INFO:teuthology.orchestra.run.smithi160.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 235 kB/s | 188 kB 00:00 2024-06-01T00:43:35.027 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:35.027 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.2 MB/s | 3.8 MB 00:01 2024-06-01T00:43:35.027 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:43:35.035 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:43:35.035 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:43:35.047 INFO:teuthology.orchestra.run.smithi079.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1387.g71c5cfc9 5.4 MB/s | 3.5 MB 00:00 2024-06-01T00:43:35.051 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:35.051 INFO:teuthology.orchestra.run.smithi079.stdout:Total 4.7 MB/s | 3.8 MB 00:00 2024-06-01T00:43:35.051 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:35.058 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:35.058 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:35.098 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:43:35.098 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:43:35.118 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:35.118 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:35.428 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:43:35.430 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:35.507 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-06-01T00:43:35.522 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-06-01T00:43:35.546 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-06-01T00:43:35.552 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-06-01T00:43:35.575 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1387.g71c5cfc9.el9.n 3/5 2024-06-01T00:43:35.581 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1387.g71c5cfc9.el9.n 3/5 2024-06-01T00:43:36.617 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1387.g71c5cfc9.el9. 4/5 2024-06-01T00:43:36.624 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1387.g71c5cfc9.el9. 4/5 2024-06-01T00:43:36.654 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarc 5/5 2024-06-01T00:43:36.660 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarc 5/5 2024-06-01T00:43:37.718 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarc 5/5 2024-06-01T00:43:37.718 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1387.g71c5cfc9.el9. 1/5 2024-06-01T00:43:37.719 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarc 2/5 2024-06-01T00:43:37.719 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1387.g71c5cfc9.el9.n 3/5 2024-06-01T00:43:37.719 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-06-01T00:43:37.765 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarc 5/5 2024-06-01T00:43:37.765 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1387.g71c5cfc9.el9. 1/5 2024-06-01T00:43:37.765 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarc 2/5 2024-06-01T00:43:37.765 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1387.g71c5cfc9.el9.n 3/5 2024-06-01T00:43:37.765 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-06-01T00:43:37.967 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-06-01T00:43:37.967 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:37.967 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:43:37.967 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:37.968 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:37.968 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:37.968 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-01T00:43:37.968 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-01T00:43:37.968 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:37.968 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:43:38.014 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:38.015 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:38.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-01T00:43:38.228 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-01T00:43:38.737 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:07 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:43:38.803 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:01 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:38.861 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:43:38.862 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-06-01T00:43:38.862 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:43:38.862 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 7.4 M 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-01T00:43:38.863 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout:Install 12 Packages 2024-06-01T00:43:38.864 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:43:38.865 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 38 M 2024-06-01T00:43:38.865 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 209 M 2024-06-01T00:43:38.865 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:43:38.925 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:38.926 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-06-01T00:43:38.926 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:43:38.926 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-06-01T00:43:38.926 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:38.926 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 7.4 M 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:38.927 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:38.928 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-06-01T00:43:38.928 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-06-01T00:43:38.928 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:38.928 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 38 M 2024-06-01T00:43:38.928 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 209 M 2024-06-01T00:43:38.928 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:39.123 INFO:teuthology.orchestra.run.smithi160.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-06-01T00:43:39.157 INFO:teuthology.orchestra.run.smithi160.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.2 MB/s | 803 kB 00:00 2024-06-01T00:43:39.390 INFO:teuthology.orchestra.run.smithi160.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 122 kB/s | 33 kB 00:00 2024-06-01T00:43:39.507 INFO:teuthology.orchestra.run.smithi160.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 156 kB/s | 18 kB 00:00 2024-06-01T00:43:39.512 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 382 kB/s | 195 kB 00:00 2024-06-01T00:43:39.616 INFO:teuthology.orchestra.run.smithi160.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-06-01T00:43:39.621 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): ceph-mgr-diskprediction-local-18.2.2-13 12 MB/s | 7.4 MB 00:00 2024-06-01T00:43:39.688 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 187 kB/s | 33 kB 00:00 2024-06-01T00:43:39.730 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 435 kB/s | 18 kB 00:00 2024-06-01T00:43:39.781 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-06-01T00:43:39.805 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-06-01T00:43:39.947 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.2 MB/s | 3.0 MB 00:00 2024-06-01T00:43:40.016 INFO:teuthology.orchestra.run.smithi160.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.2-13 7.1 MB/s | 7.4 MB 00:01 2024-06-01T00:43:40.048 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.0 MB/s | 244 kB 00:00 2024-06-01T00:43:40.150 INFO:teuthology.orchestra.run.smithi160.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:00 2024-06-01T00:43:40.165 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-06-01T00:43:40.232 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.3 MB/s | 250 kB 00:00 2024-06-01T00:43:40.457 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-06-01T00:43:40.542 INFO:teuthology.orchestra.run.smithi160.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-06-01T00:43:40.616 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-06-01T00:43:40.617 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:40.617 INFO:teuthology.orchestra.run.smithi079.stdout:Total 23 MB/s | 38 MB 00:01 2024-06-01T00:43:40.700 INFO:teuthology.orchestra.run.smithi160.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 357 kB/s | 244 kB 00:00 2024-06-01T00:43:40.892 INFO:teuthology.orchestra.run.smithi160.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 715 kB/s | 250 kB 00:00 2024-06-01T00:43:40.969 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:40.996 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:40.996 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:41.500 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:41.500 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:42.251 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:42.320 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-01T00:43:42.355 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-01T00:43:42.450 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-01T00:43:42.506 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-01T00:43:42.683 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-01T00:43:42.714 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-01T00:43:42.791 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-01T00:43:43.469 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-01T00:43:43.586 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-01T00:43:44.869 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-01T00:43:45.890 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-01T00:43:45.930 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5c 12/12 2024-06-01T00:43:46.875 INFO:teuthology.orchestra.run.smithi160.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 776 kB/s | 5.1 MB 00:06 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5c 12/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5c 1/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-01T00:43:48.434 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-01T00:43:48.435 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-01T00:43:48.435 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-01T00:43:48.799 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-01T00:43:48.800 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-01T00:43:48.801 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-01T00:43:48.801 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-01T00:43:48.801 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:48.801 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:49.150 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-06-01T00:43:49.741 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:12 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:49.865 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:49.866 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:49.866 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:43:49.866 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:49.867 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:49.867 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 49 k 2024-06-01T00:43:49.867 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:43:49.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-01T00:43:49.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-01T00:43:49.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-01T00:43:49.867 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-01T00:43:49.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-01T00:43:49.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-01T00:43:49.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-01T00:43:49.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-01T00:43:49.868 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-06-01T00:43:49.869 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:49.870 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.2 M 2024-06-01T00:43:49.870 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-06-01T00:43:49.870 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:50.380 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 207 kB/s | 19 kB 00:00 2024-06-01T00:43:50.406 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 218 kB/s | 26 kB 00:00 2024-06-01T00:43:50.464 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-06-01T00:43:50.489 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): ceph-mgr-rook-18.2.2-1387.g71c5cfc9.el9 244 kB/s | 49 kB 00:00 2024-06-01T00:43:50.540 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-06-01T00:43:50.565 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 996 kB/s | 159 kB 00:00 2024-06-01T00:43:50.624 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 400 kB/s | 54 kB 00:00 2024-06-01T00:43:50.707 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-06-01T00:43:50.749 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 81 kB/s | 15 kB 00:00 2024-06-01T00:43:50.883 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 794 kB/s | 206 kB 00:00 2024-06-01T00:43:50.933 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-06-01T00:43:50.966 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-06-01T00:43:51.000 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-06-01T00:43:51.000 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:51.001 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-06-01T00:43:51.071 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:51.086 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:51.086 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:51.173 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:51.173 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:51.670 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:51.834 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-01T00:43:51.884 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-01T00:43:51.942 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-01T00:43:51.979 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-01T00:43:52.021 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-01T00:43:52.119 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-01T00:43:52.228 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-01T00:43:52.294 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-01T00:43:53.011 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-01T00:43:53.058 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-01T00:43:53.092 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-01T00:43:53.140 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-01T00:43:53.162 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 13/13 2024-06-01T00:43:53.874 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 13/13 2024-06-01T00:43:53.874 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/13 2024-06-01T00:43:53.874 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-01T00:43:53.874 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-01T00:43:53.874 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-01T00:43:53.874 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-01T00:43:53.874 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-01T00:43:53.875 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-01T00:43:53.875 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-01T00:43:53.875 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-01T00:43:53.875 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-01T00:43:53.875 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-01T00:43:53.875 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-01T00:43:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-01T00:43:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:54.195 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:54.195 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:54.196 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:54.459 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-06-01T00:43:55.071 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:18 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:43:55.200 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:43:55.201 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:55.201 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:43:55.201 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:55.201 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:43:55.201 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 139 k 2024-06-01T00:43:55.201 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-06-01T00:43:55.202 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:55.203 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 7.0 M 2024-06-01T00:43:55.203 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-06-01T00:43:55.203 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:43:55.745 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): ceph-mgr-cephadm-18.2.2-1387.g71c5cfc9.e 435 kB/s | 139 kB 00:00 2024-06-01T00:43:55.920 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 503 kB/s | 248 kB 00:00 2024-06-01T00:43:56.387 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 124 kB/s | 58 kB 00:00 2024-06-01T00:43:56.487 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 739 kB/s | 548 kB 00:00 2024-06-01T00:43:56.529 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 359 kB/s | 51 kB 00:00 2024-06-01T00:43:56.987 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.8 MB/s | 6.0 MB 00:01 2024-06-01T00:43:56.990 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:43:56.990 INFO:teuthology.orchestra.run.smithi079.stdout:Total 3.9 MB/s | 7.0 MB 00:01 2024-06-01T00:43:57.061 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:43:57.073 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:43:57.073 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:43:57.170 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:43:57.172 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:43:57.476 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:43:57.653 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-01T00:43:57.707 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-01T00:43:58.215 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-01T00:43:58.301 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-01T00:43:58.363 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-01T00:43:58.386 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 6/6 2024-06-01T00:43:59.149 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 6/6 2024-06-01T00:43:59.149 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/6 2024-06-01T00:43:59.149 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-01T00:43:59.149 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-01T00:43:59.149 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-01T00:43:59.149 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-01T00:43:59.409 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:43:59.410 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:43:59.652 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-06-01T00:44:00.229 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:23 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:00.351 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 849 k 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-01T00:44:00.352 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:00.353 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:44:00.353 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:00.353 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-06-01T00:44:00.353 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:00.353 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 929 k 2024-06-01T00:44:00.353 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 2.7 M 2024-06-01T00:44:00.353 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:44:00.666 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): ceph-fuse-18.2.2-1387.g71c5cfc9.el9.x86_ 3.0 MB/s | 849 kB 00:00 2024-06-01T00:44:00.875 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 166 kB/s | 80 kB 00:00 2024-06-01T00:44:00.875 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:00.875 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.7 MB/s | 929 kB 00:00 2024-06-01T00:44:00.887 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:44:00.909 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:44:00.909 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:44:00.986 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:44:00.986 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:44:01.176 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:44:01.307 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-01T00:44:01.374 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/2 2024-06-01T00:44:02.010 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/2 2024-06-01T00:44:02.010 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/2 2024-06-01T00:44:02.312 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-01T00:44:02.312 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:02.312 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:44:02.312 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-01T00:44:02.312 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:02.312 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:02.348 INFO:teuthology.orchestra.run.smithi160.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 769 kB/s | 16 MB 00:21 2024-06-01T00:44:02.349 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:02.350 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.6 MB/s | 38 MB 00:23 2024-06-01T00:44:02.556 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-06-01T00:44:02.703 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:02.730 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:02.730 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:03.134 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:26 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:03.246 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:03.246 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:03.259 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:03.259 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-06-01T00:44:03.259 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:03.259 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:44:03.259 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 127 k 2024-06-01T00:44:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:03.260 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:44:03.260 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:03.260 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-06-01T00:44:03.260 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:03.260 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 127 k 2024-06-01T00:44:03.260 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 456 k 2024-06-01T00:44:03.261 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:44:03.565 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-18.2.2-1387.g71c5cfc9.el9.x86_64 416 kB/s | 127 kB 00:00 2024-06-01T00:44:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:Total 414 kB/s | 127 kB 00:00 2024-06-01T00:44:03.566 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:44:03.571 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:44:03.571 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:44:03.633 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:44:03.634 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:44:03.785 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:44:03.895 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:04.006 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:04.077 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-01T00:44:04.121 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-01T00:44:04.221 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-01T00:44:04.256 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-01T00:44:04.299 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:04.435 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-01T00:44:04.471 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-01T00:44:04.541 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:04.541 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:04.541 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:44:04.542 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:04.542 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:04.542 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:04.548 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-01T00:44:04.771 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-06-01T00:44:05.194 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-01T00:44:05.318 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-01T00:44:05.384 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:28 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:05.468 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:05.511 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:05.511 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:44:05.512 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:05.589 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-06-01T00:44:06.174 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:29 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:06.299 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 31 k 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:06.300 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-06-01T00:44:06.301 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:06.301 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 31 k 2024-06-01T00:44:06.301 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 139 k 2024-06-01T00:44:06.301 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:44:06.479 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-18.2.2-1387.g71c5cfc9.el9.x86_6 174 kB/s | 31 kB 00:00 2024-06-01T00:44:06.479 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:06.480 INFO:teuthology.orchestra.run.smithi079.stdout:Total 173 kB/s | 31 kB 00:00 2024-06-01T00:44:06.480 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:44:06.483 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:44:06.483 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:44:06.501 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:44:06.502 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:44:06.574 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-01T00:44:06.582 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:44:06.675 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:07.033 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:07.257 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:07.258 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:07.258 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:44:07.258 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:07.258 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:07.258 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:07.445 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-06-01T00:44:07.616 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-01T00:44:07.652 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5c 12/12 2024-06-01T00:44:08.026 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:31 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:08.109 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:08.147 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:08.148 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:44:08.149 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:08.224 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-06-01T00:44:08.797 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:31 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:08.877 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:08.919 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:08.920 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:44:08.920 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:09.001 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-06-01T00:44:09.579 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:32 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:09.662 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:09.704 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:09.705 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:44:09.705 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:09.782 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5c 12/12 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5c 1/12 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-01T00:44:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-01T00:44:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-01T00:44:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-01T00:44:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-01T00:44:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-01T00:44:10.362 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:33 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:10.439 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-01T00:44:10.439 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-01T00:44:10.440 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-01T00:44:10.441 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-01T00:44:10.441 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:10.441 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:10.441 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:10.483 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:10.484 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:44:10.484 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:10.553 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-06-01T00:44:10.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2024-06-01T00:44:11.128 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:34 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:11.210 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:11.253 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:11.253 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:44:11.254 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:11.321 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-06-01T00:44:11.405 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:40 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:11.531 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:11.532 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 49 k 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-01T00:44:11.533 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2024-06-01T00:44:11.534 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:11.535 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 2.2 M 2024-06-01T00:44:11.535 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 26 M 2024-06-01T00:44:11.535 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:11.839 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): ceph-mgr-rook-18.2.2-1387.g71c5cfc9.el9 252 kB/s | 49 kB 00:00 2024-06-01T00:44:11.902 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:34 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:11.906 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 97 kB/s | 26 kB 00:00 2024-06-01T00:44:11.932 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 66 kB/s | 19 kB 00:00 2024-06-01T00:44:11.984 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:12.025 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:12.026 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-06-01T00:44:12.026 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:12.098 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-06-01T00:44:12.248 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-06-01T00:44:12.369 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 447 kB/s | 54 kB 00:00 2024-06-01T00:44:12.394 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 603 kB/s | 279 kB 00:00 2024-06-01T00:44:12.519 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 327 kB/s | 222 kB 00:00 2024-06-01T00:44:12.586 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-06-01T00:44:12.612 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-06-01T00:44:12.670 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-06-01T00:44:12.679 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:35 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:12.729 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-06-01T00:44:12.754 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-google-auth-2.29.0-1.el9.noarc 880 kB/s | 206 kB 00:00 2024-06-01T00:44:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:12.802 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 87 k 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 87 k 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 227 k 2024-06-01T00:44:12.803 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:44:12.804 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-06-01T00:44:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:12.807 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-06-01T00:44:12.870 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:12.884 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:12.885 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:12.971 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:12.972 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:12.991 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-18.2.2-1387.g71c5cfc9.el9.x86_64.rpm 464 kB/s | 87 kB 00:00 2024-06-01T00:44:12.991 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:12.991 INFO:teuthology.orchestra.run.smithi079.stdout:Total 459 kB/s | 87 kB 00:00 2024-06-01T00:44:12.992 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:44:12.998 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:44:12.999 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:44:13.060 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:44:13.061 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:44:13.226 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:44:13.321 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:13.496 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-01T00:44:13.712 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-01T00:44:13.777 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-01T00:44:13.815 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-01T00:44:13.856 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-01T00:44:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-01T00:44:13.973 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:13.973 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:13.973 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:44:13.973 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:13.973 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:13.973 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-01T00:44:14.125 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-01T00:44:14.202 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-06-01T00:44:14.789 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:37 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:14.851 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-01T00:44:14.905 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-01T00:44:14.917 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:14.917 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:14.917 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 3.0 M 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 3.0 M 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 12 M 2024-06-01T00:44:14.918 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:44:14.939 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-01T00:44:14.986 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-01T00:44:15.008 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 13/13 2024-06-01T00:44:15.305 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-18.2.2-1387.g71c5cfc9.el9.x86_64.rpm 7.7 MB/s | 3.0 MB 00:00 2024-06-01T00:44:15.305 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:15.305 INFO:teuthology.orchestra.run.smithi079.stdout:Total 7.7 MB/s | 3.0 MB 00:00 2024-06-01T00:44:15.306 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:44:15.315 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:44:15.316 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:44:15.393 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:44:15.393 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 13/13 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/13 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-01T00:44:15.723 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-01T00:44:15.724 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-01T00:44:15.724 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-01T00:44:15.724 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-01T00:44:15.724 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-01T00:44:15.724 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-01T00:44:15.807 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:44:15.863 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:15.888 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:15.888 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:44:15.888 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-01T00:44:15.888 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-01T00:44:15.888 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-01T00:44:15.888 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:16.037 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-01T00:44:16.037 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:16.037 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:16.037 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:44:16.037 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-01T00:44:16.037 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-01T00:44:16.037 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:16.038 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:16.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2024-06-01T00:44:16.892 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:45 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:16.913 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:16.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:16.914 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:44:16.914 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:16.914 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:16.914 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:17.017 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1387.g71c5cfc9.el9 ceph-noarch 139 k 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-01T00:44:17.018 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 7.0 M 2024-06-01T00:44:17.019 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 33 M 2024-06-01T00:44:17.020 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:17.153 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-06-01T00:44:17.457 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-mgr-cephadm-18.2.2-1387.g71c5cfc9.e 566 kB/s | 139 kB 00:00 2024-06-01T00:44:17.744 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:40 ago on Sat 01 Jun 2024 12:42:37 AM UTC. 2024-06-01T00:44:17.799 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 423 kB/s | 248 kB 00:00 2024-06-01T00:44:17.870 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-06-01T00:44:17.871 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:17.871 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-06-01T00:44:17.871 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:17.871 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-06-01T00:44:17.871 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 172 k 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 172 k 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 494 k 2024-06-01T00:44:17.872 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-06-01T00:44:18.082 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 204 kB/s | 58 kB 00:00 2024-06-01T00:44:18.241 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 699 kB/s | 548 kB 00:00 2024-06-01T00:44:18.266 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2024-06-01T00:44:18.318 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-18.2.2-1387.g71c5cfc9.el9.x86_64.rpm 386 kB/s | 172 kB 00:00 2024-06-01T00:44:18.318 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:18.319 INFO:teuthology.orchestra.run.smithi079.stdout:Total 384 kB/s | 172 kB 00:00 2024-06-01T00:44:18.319 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-06-01T00:44:18.327 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-06-01T00:44:18.327 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-06-01T00:44:18.390 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-06-01T00:44:18.390 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-06-01T00:44:18.541 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-06-01T00:44:18.543 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:18.543 INFO:teuthology.orchestra.run.smithi160.stdout:Total 4.6 MB/s | 7.0 MB 00:01 2024-06-01T00:44:18.553 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-06-01T00:44:18.615 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:18.626 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:18.627 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:18.657 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:18.725 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:18.725 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:19.044 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:19.109 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:19.206 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-01T00:44:19.268 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-01T00:44:19.361 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:19.361 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:19.361 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-06-01T00:44:19.361 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:19.361 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:44:19.361 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-06-01T00:44:19.581 DEBUG:teuthology.parallel:result is None 2024-06-01T00:44:19.799 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-01T00:44:19.879 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-01T00:44:19.941 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-01T00:44:19.972 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 6/6 2024-06-01T00:44:20.758 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 6/6 2024-06-01T00:44:20.758 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 1/6 2024-06-01T00:44:20.758 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-01T00:44:20.758 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-01T00:44:20.759 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-01T00:44:20.759 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:18.2.2-1387.g71c5cfc9.el9.noarch 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-01T00:44:21.036 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:21.037 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:21.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2024-06-01T00:44:21.869 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:50 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:21.993 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 849 k 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-01T00:44:21.994 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:21.995 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:21.995 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:21.995 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2024-06-01T00:44:21.995 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:21.995 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 929 k 2024-06-01T00:44:21.995 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 2.7 M 2024-06-01T00:44:21.995 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:22.418 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): ceph-fuse-18.2.2-1387.g71c5cfc9.el9.x86_ 2.7 MB/s | 849 kB 00:00 2024-06-01T00:44:22.543 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 188 kB/s | 80 kB 00:00 2024-06-01T00:44:22.543 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:22.544 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.7 MB/s | 929 kB 00:00 2024-06-01T00:44:22.555 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:22.577 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:22.577 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:22.656 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:22.657 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:22.838 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:22.967 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-01T00:44:23.034 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/2 2024-06-01T00:44:23.700 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2/2 2024-06-01T00:44:23.700 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/2 2024-06-01T00:44:23.944 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-01T00:44:23.944 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:23.944 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:23.944 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-01T00:44:23.944 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:23.944 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:24.180 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2024-06-01T00:44:24.769 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:53 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:24.898 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:24.899 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:24.899 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 127 k 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 127 k 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 456 k 2024-06-01T00:44:24.900 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:25.122 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-18.2.2-1387.g71c5cfc9.el9.x86_64 570 kB/s | 127 kB 00:00 2024-06-01T00:44:25.123 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:25.123 INFO:teuthology.orchestra.run.smithi160.stdout:Total 566 kB/s | 127 kB 00:00 2024-06-01T00:44:25.123 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:25.129 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:25.129 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:25.191 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:25.192 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:25.336 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:25.469 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:25.892 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:26.127 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:26.127 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:26.127 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:26.128 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:26.128 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:26.128 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:26.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2024-06-01T00:44:26.955 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:55 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:27.038 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:27.080 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:27.081 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:44:27.081 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:27.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2024-06-01T00:44:27.734 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:56 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:27.861 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 31 k 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:27.862 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-06-01T00:44:27.863 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:27.863 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 31 k 2024-06-01T00:44:27.863 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 139 k 2024-06-01T00:44:27.863 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:28.101 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-18.2.2-1387.g71c5cfc9.el9.x86_6 130 kB/s | 31 kB 00:00 2024-06-01T00:44:28.101 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:28.101 INFO:teuthology.orchestra.run.smithi160.stdout:Total 130 kB/s | 31 kB 00:00 2024-06-01T00:44:28.102 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:28.105 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:28.105 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:28.116 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:28.116 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:28.200 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:28.324 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:28.700 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:28.926 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:28.926 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:28.927 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:28.927 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:29.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2024-06-01T00:44:29.703 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:58 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:29.785 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:29.827 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:29.828 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:44:29.828 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:29.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2024-06-01T00:44:30.476 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:59 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:30.557 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:30.599 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:30.600 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:44:30.600 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:30.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2024-06-01T00:44:31.255 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:00 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:31.338 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:31.381 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:31.382 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:44:31.382 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:31.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2024-06-01T00:44:32.034 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:01 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:32.116 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:32.158 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:32.159 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:44:32.159 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:32.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2024-06-01T00:44:32.799 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:01 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:32.882 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:32.925 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:32.926 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:44:32.926 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:32.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2024-06-01T00:44:33.576 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:02 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:33.658 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 is already installed. 2024-06-01T00:44:33.701 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:33.702 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2024-06-01T00:44:33.702 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:33.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2024-06-01T00:44:34.352 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:03 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:34.475 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:34.476 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:34.476 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-06-01T00:44:34.476 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:34.476 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 87 k 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 87 k 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 227 k 2024-06-01T00:44:34.477 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:34.787 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-18.2.2-1387.g71c5cfc9.el9.x86_64.rpm 279 kB/s | 87 kB 00:00 2024-06-01T00:44:34.787 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:34.788 INFO:teuthology.orchestra.run.smithi160.stdout:Total 278 kB/s | 87 kB 00:00 2024-06-01T00:44:34.788 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:34.795 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:34.795 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:34.858 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:34.859 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:35.008 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:35.108 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:35.485 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:35.738 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:35.738 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:35.738 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:35.738 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:35.738 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:35.738 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:36.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2024-06-01T00:44:36.597 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:05 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:36.725 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:36.726 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:36.726 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2024-06-01T00:44:36.726 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:36.726 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 3.0 M 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.0 M 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 12 M 2024-06-01T00:44:36.727 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:37.098 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-18.2.2-1387.g71c5cfc9.el9.x86_64.rpm 8.0 MB/s | 3.0 MB 00:00 2024-06-01T00:44:37.099 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:37.099 INFO:teuthology.orchestra.run.smithi160.stdout:Total 8.0 MB/s | 3.0 MB 00:00 2024-06-01T00:44:37.099 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:37.108 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:37.108 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:37.187 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:37.188 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:37.604 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:37.666 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:37.690 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:37.690 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-01T00:44:37.690 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-01T00:44:37.690 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-01T00:44:37.690 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-01T00:44:37.690 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:38.686 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:38.686 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:38.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2024-06-01T00:44:39.528 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:08 ago on Sat 01 Jun 2024 12:42:31 AM UTC. 2024-06-01T00:44:39.657 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:18.2.2-1387.g71c5cfc9.el9 ceph 172 k 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 172 k 2024-06-01T00:44:39.658 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 494 k 2024-06-01T00:44:39.659 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-06-01T00:44:39.886 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-18.2.2-1387.g71c5cfc9.el9.x86_64.rpm 758 kB/s | 172 kB 00:00 2024-06-01T00:44:39.887 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:44:39.887 INFO:teuthology.orchestra.run.smithi160.stdout:Total 751 kB/s | 172 kB 00:00 2024-06-01T00:44:39.887 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-06-01T00:44:39.895 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-06-01T00:44:39.895 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-06-01T00:44:39.961 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-06-01T00:44:39.962 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-06-01T00:44:40.117 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-06-01T00:44:40.233 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:40.702 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:40.946 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 1/1 2024-06-01T00:44:40.946 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:40.946 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-06-01T00:44:40.947 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:18.2.2-1387.g71c5cfc9.el9.x86_64 2024-06-01T00:44:40.947 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:44:40.947 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-06-01T00:44:41.178 DEBUG:teuthology.parallel:result is None 2024-06-01T00:44:41.179 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T00:44:41.179 INFO:teuthology.packaging:ref: None 2024-06-01T00:44:41.179 INFO:teuthology.packaging:tag: None 2024-06-01T00:44:41.179 INFO:teuthology.packaging:branch: reef 2024-06-01T00:44:41.179 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:44:41.180 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-01T00:44:41.527 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-01T00:44:41.553 INFO:teuthology.orchestra.run.smithi079.stdout:18.2.2-1387.g71c5cfc9.el9 2024-06-01T00:44:41.554 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1387.g71c5cfc9.el9 2024-06-01T00:44:41.554 INFO:teuthology.task.install:The correct ceph version 18.2.2-1387.g71c5cfc9 is installed. 2024-06-01T00:44:41.556 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T00:44:41.556 INFO:teuthology.packaging:ref: None 2024-06-01T00:44:41.556 INFO:teuthology.packaging:tag: None 2024-06-01T00:44:41.556 INFO:teuthology.packaging:branch: reef 2024-06-01T00:44:41.556 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:44:41.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-01T00:44:41.806 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-01T00:44:41.830 INFO:teuthology.orchestra.run.smithi160.stdout:18.2.2-1387.g71c5cfc9.el9 2024-06-01T00:44:41.830 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1387.g71c5cfc9.el9 2024-06-01T00:44:41.830 INFO:teuthology.task.install:The correct ceph version 18.2.2-1387.g71c5cfc9 is installed. 2024-06-01T00:44:41.832 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-01T00:44:41.832 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:44:41.832 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-01T00:44:41.863 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:44:41.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-01T00:44:41.898 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-01T00:44:41.898 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:44:41.898 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-06-01T00:44:41.932 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-01T00:44:42.000 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:44:42.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2024-06-01T00:44:42.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-01T00:44:42.094 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-01T00:44:42.094 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:44:42.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-01T00:44:42.122 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-01T00:44:42.192 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:44:42.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-01T00:44:42.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-01T00:44:42.286 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-01T00:44:42.286 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:44:42.286 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-06-01T00:44:42.311 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-01T00:44:42.377 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:44:42.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2024-06-01T00:44:42.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-01T00:44:42.468 INFO:teuthology.run_tasks:Running task print... 2024-06-01T00:44:42.477 INFO:teuthology.task.print:**** done install task... 2024-06-01T00:44:42.478 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-01T00:44:42.574 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-01T00:44:42.575 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-06-01T00:44:42.575 INFO:tasks.cephadm:Cluster fsid is 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:44:42.575 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-01T00:44:42.576 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-01T00:44:42.576 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi079': '172.21.15.79', 'mon.smithi160': '172.21.15.160'} 2024-06-01T00:44:42.576 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-01T00:44:42.576 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-06-01T00:44:42.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2024-06-01T00:44:42.626 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-06-01T00:44:42.626 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T00:44:42.844 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55273/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2946-g3d89accf', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2946.g3d89accf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-30 17:04:17.570799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-06-01T00:44:43.091 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref reef, sha1 71c5cfc99f93c162032256d34d9b60b8288fad91 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-06-01T00:44:43.093 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/x86_64/flavors/default/cephadm 2024-06-01T00:44:43.093 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-01T00:44:43.641 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 215335 Jun 1 00:44 /home/ubuntu/cephtest/cephadm 2024-06-01T00:44:43.641 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/reef/71c5cfc99f93c162032256d34d9b60b8288fad91/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-01T00:44:44.063 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-r--r--. 1 ubuntu ubuntu 215335 Jun 1 00:44 /home/ubuntu/cephtest/cephadm 2024-06-01T00:44:44.063 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-01T00:44:44.081 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-01T00:44:44.111 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-06-01T00:44:44.112 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-06-01T00:44:44.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-06-01T00:44:44.910 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-06-01T00:44:44.985 INFO:teuthology.orchestra.run.smithi160.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-06-01T00:45:12.569 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-06-01T00:45:12.569 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_version": "ceph version 18.2.2-1387-g71c5cfc9 (71c5cfc99f93c162032256d34d9b60b8288fad91) reef (stable)", 2024-06-01T00:45:12.569 INFO:teuthology.orchestra.run.smithi160.stdout: "image_id": "53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe", 2024-06-01T00:45:12.570 INFO:teuthology.orchestra.run.smithi160.stdout: "repo_digests": [ 2024-06-01T00:45:12.570 INFO:teuthology.orchestra.run.smithi160.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58" 2024-06-01T00:45:12.570 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-06-01T00:45:12.570 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-06-01T00:45:13.932 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-06-01T00:45:13.933 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 18.2.2-1387-g71c5cfc9 (71c5cfc99f93c162032256d34d9b60b8288fad91) reef (stable)", 2024-06-01T00:45:13.933 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe", 2024-06-01T00:45:13.933 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-06-01T00:45:13.933 INFO:teuthology.orchestra.run.smithi079.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58" 2024-06-01T00:45:13.933 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-06-01T00:45:13.933 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-06-01T00:45:13.951 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-06-01T00:45:14.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2024-06-01T00:45:14.052 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-06-01T00:45:14.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2024-06-01T00:45:14.120 INFO:tasks.cephadm:Writing seed config... 2024-06-01T00:45:14.122 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-01T00:45:14.122 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-01T00:45:14.122 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-01T00:45:14.122 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-01T00:45:14.122 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-01T00:45:14.123 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-01T00:45:14.124 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-01T00:45:14.125 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-01T00:45:14.126 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:45:14.126 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-01T00:45:14.142 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 22870f5e-1fb0-11ef-bc9b-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-01T00:45:14.142 DEBUG:teuthology.orchestra.run.smithi079:mon.smithi079> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079.service 2024-06-01T00:45:14.185 INFO:tasks.cephadm:Bootstrapping... 2024-06-01T00:45:14.185 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.79 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-01T00:45:14.339 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-06-01T00:45:14.339 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '22870f5e-1fb0-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.79', '--skip-admin-label'] 2024-06-01T00:45:14.364 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.0.2 2024-06-01T00:45:14.365 INFO:teuthology.orchestra.run.smithi079.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-01T00:45:14.365 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying podman|docker is present... 2024-06-01T00:45:14.388 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.0.2 2024-06-01T00:45:14.388 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying lvm2 is present... 2024-06-01T00:45:14.389 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying time synchronization is in place... 2024-06-01T00:45:14.395 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-01T00:45:14.396 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-01T00:45:14.401 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-01T00:45:14.402 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-06-01T00:45:14.408 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-06-01T00:45:14.414 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-06-01T00:45:14.414 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-06-01T00:45:14.414 INFO:teuthology.orchestra.run.smithi079.stdout:Repeating the final host check... 2024-06-01T00:45:14.437 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.0.2 2024-06-01T00:45:14.437 INFO:teuthology.orchestra.run.smithi079.stdout:podman (/bin/podman) version 5.0.2 is present 2024-06-01T00:45:14.437 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl is present 2024-06-01T00:45:14.437 INFO:teuthology.orchestra.run.smithi079.stdout:lvcreate is present 2024-06-01T00:45:14.444 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-01T00:45:14.444 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-01T00:45:14.449 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-01T00:45:14.450 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-06-01T00:45:14.456 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-06-01T00:45:14.462 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-06-01T00:45:14.462 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-06-01T00:45:14.463 INFO:teuthology.orchestra.run.smithi079.stdout:Host looks OK 2024-06-01T00:45:14.463 INFO:teuthology.orchestra.run.smithi079.stdout:Cluster fsid: 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:14.463 INFO:teuthology.orchestra.run.smithi079.stdout:Acquiring lock 140553687482672 on /run/cephadm/22870f5e-1fb0-11ef-bc9b-c7b262605968.lock 2024-06-01T00:45:14.463 INFO:teuthology.orchestra.run.smithi079.stdout:Lock 140553687482672 acquired on /run/cephadm/22870f5e-1fb0-11ef-bc9b-c7b262605968.lock 2024-06-01T00:45:14.463 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 3300 ... 2024-06-01T00:45:14.464 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 6789 ... 2024-06-01T00:45:14.464 INFO:teuthology.orchestra.run.smithi079.stdout:Base mon IP(s) is [172.21.15.79:3300, 172.21.15.79:6789], mon addrv is [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-06-01T00:45:14.466 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.79 metric 100 2024-06-01T00:45:14.466 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.79 metric 100 2024-06-01T00:45:14.469 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-01T00:45:14.469 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-01T00:45:14.469 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-06-01T00:45:14.471 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-01T00:45:14.471 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-01T00:45:14.471 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-01T00:45:14.471 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-01T00:45:14.472 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce87/64 scope link 2024-06-01T00:45:14.472 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-01T00:45:14.472 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-06-01T00:45:14.472 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-06-01T00:45:14.472 INFO:teuthology.orchestra.run.smithi079.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-01T00:45:14.473 INFO:teuthology.orchestra.run.smithi079.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-01T00:45:14.473 INFO:teuthology.orchestra.run.smithi079.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-06-01T00:45:14.936 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe 2024-06-01T00:45:14.936 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-06-01T00:45:14.936 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Getting image source signatures 2024-06-01T00:45:14.936 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:50b236ea8b5f690a9a391951d9bffb2ddb93f02f3903b8c6fc29779407a85ab3 2024-06-01T00:45:14.936 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:36850128dbaa55a7fd38bf12640a70ab57c121fdc76af02f72b423e059979691 2024-06-01T00:45:14.936 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying config sha256:53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe 2024-06-01T00:45:14.937 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-01T00:45:16.118 INFO:teuthology.orchestra.run.smithi079.stdout:ceph: stdout ceph version 18.2.2-1387-g71c5cfc9 (71c5cfc99f93c162032256d34d9b60b8288fad91) reef (stable) 2024-06-01T00:45:16.118 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph version: ceph version 18.2.2-1387-g71c5cfc9 (71c5cfc99f93c162032256d34d9b60b8288fad91) reef (stable) 2024-06-01T00:45:16.118 INFO:teuthology.orchestra.run.smithi079.stdout:Extracting ceph user uid/gid from container image... 2024-06-01T00:45:17.023 INFO:teuthology.orchestra.run.smithi079.stdout:stat: stdout 167 167 2024-06-01T00:45:17.023 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial keys... 2024-06-01T00:45:17.946 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQAdb1pmcpAkFRAAJBb4UJaV1GzuRVaIqfWLtQ== 2024-06-01T00:45:18.894 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQAeb1pm59ydERAA3QwnWmICg0OcjGORiX0LNA== 2024-06-01T00:45:19.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQAfb1pmaxCIEBAAXhdmcvxuWgASz/Ftlrh6Gg== 2024-06-01T00:45:19.826 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial monmap... 2024-06-01T00:45:20.936 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-01T00:45:20.936 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-06-01T00:45:20.936 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:20.936 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-01T00:45:20.937 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool for smithi079 [v2:172.21.15.79:3300,v1:172.21.15.79:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-01T00:45:20.937 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = pacific 2024-06-01T00:45:20.937 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: set fsid to 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:20.937 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-01T00:45:20.937 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:45:20.937 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mon... 2024-06-01T00:45:30.077 INFO:teuthology.orchestra.run.smithi079.stdout:create mon.smithi079 on 2024-06-01T00:45:30.604 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-01T00:45:31.022 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-01T00:45:31.204 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968.target. 2024-06-01T00:45:31.204 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968.target. 2024-06-01T00:45:31.429 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079 2024-06-01T00:45:31.429 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079.service: Unit ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079.service not loaded. 2024-06-01T00:45:31.588 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968.target.wants/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079.service → /etc/systemd/system/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@.service. 2024-06-01T00:45:32.077 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:32 smithi079 ceph-mon[28986]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-06-01T00:45:32.135 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-01T00:45:32.135 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-06-01T00:45:32.143 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-01T00:45:32.143 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-06-01T00:45:32.143 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-06-01T00:45:32.143 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-06-01T00:45:32.143 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon to start... 2024-06-01T00:45:32.143 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon... 2024-06-01T00:45:32.345 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:32 smithi079 bash[28966]: b5b85d688ba4d29b523878aa47a018a13b92847da74e362760af515518e5d864 2024-06-01T00:45:32.345 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:32 smithi079 systemd[1]: Started Ceph mon.smithi079 for 22870f5e-1fb0-11ef-bc9b-c7b262605968. 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout cluster: 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout id: 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout services: 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi079 (age 0.532194s) 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout data: 2024-06-01T00:45:33.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pgs: 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.533+0000 7fe06ea24640 1 Processor -- start 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.533+0000 7fe06ea24640 1 -- start start 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.533+0000 7fe06ea24640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068106080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.533+0000 7fe06ea24640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068106650 con 0x7fe068105c60 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.533+0000 7fe067fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.533+0000 7fe067fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068106080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46574/0 (socket says 172.21.15.79:46574) 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.533+0000 7fe067fff640 1 -- 172.21.15.79:0/3727221900 learned_addr learned my addr 172.21.15.79:0/3727221900 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:33.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.534+0000 7fe067fff640 1 -- 172.21.15.79:0/3727221900 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068106e60 con 0x7fe068105c60 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe067fff640 1 --2- 172.21.15.79:0/3727221900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068106080 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe058009820 tx=0x7fe05802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36db4d2193dd2bc0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe066ffd640 1 -- 172.21.15.79:0/3727221900 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe05802ad40 con 0x7fe068105c60 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe066ffd640 1 -- 172.21.15.79:0/3727221900 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe058032440 con 0x7fe068105c60 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe066ffd640 1 -- 172.21.15.79:0/3727221900 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe05802aea0 con 0x7fe068105c60 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe06ea24640 1 -- 172.21.15.79:0/3727221900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 msgr2=0x7fe068106080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe06ea24640 1 --2- 172.21.15.79:0/3727221900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068106080 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe058009820 tx=0x7fe05802a270 comp rx=0 tx=0).stop 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe06ea24640 1 -- 172.21.15.79:0/3727221900 shutdown_connections 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe06ea24640 1 --2- 172.21.15.79:0/3727221900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068106080 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.536+0000 7fe06ea24640 1 -- 172.21.15.79:0/3727221900 >> 172.21.15.79:0/3727221900 conn(0x7fe0681011c0 msgr2=0x7fe068103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.537+0000 7fe06ea24640 1 -- 172.21.15.79:0/3727221900 shutdown_connections 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.537+0000 7fe06ea24640 1 -- 172.21.15.79:0/3727221900 wait complete. 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.537+0000 7fe06ea24640 1 Processor -- start 2024-06-01T00:45:33.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.537+0000 7fe06ea24640 1 -- start start 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.537+0000 7fe06ea24640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.537+0000 7fe06ea24640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0681996a0 con 0x7fe068105c60 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe067fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe067fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46576/0 (socket says 172.21.15.79:46576) 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe067fff640 1 -- 172.21.15.79:0/3429620160 learned_addr learned my addr 172.21.15.79:0/3429620160 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe067fff640 1 -- 172.21.15.79:0/3429620160 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0580094d0 con 0x7fe068105c60 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe067fff640 1 --2- 172.21.15.79:0/3429620160 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068199160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe0580090b0 tx=0x7fe0580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe0657fa640 1 -- 172.21.15.79:0/3429620160 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0580308d0 con 0x7fe068105c60 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe0657fa640 1 -- 172.21.15.79:0/3429620160 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe058030a30 con 0x7fe068105c60 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0681998a0 con 0x7fe068105c60 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe0657fa640 1 -- 172.21.15.79:0/3429620160 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe058030d30 con 0x7fe068105c60 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe068199d40 con 0x7fe068105c60 2024-06-01T00:45:33.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe0657fa640 1 -- 172.21.15.79:0/3429620160 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe058039040 con 0x7fe068105c60 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.538+0000 7fe0657fa640 1 -- 172.21.15.79:0/3429620160 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe05803b880 con 0x7fe068105c60 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.539+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe068192dd0 con 0x7fe068105c60 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.541+0000 7fe0657fa640 1 -- 172.21.15.79:0/3429620160 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fe058037070 con 0x7fe068105c60 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.603+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe06802c410 con 0x7fe068105c60 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.603+0000 7fe0657fa640 1 -- 172.21.15.79:0/3429620160 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe05803bb40 con 0x7fe068105c60 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.604+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 msgr2=0x7fe068199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.604+0000 7fe06ea24640 1 --2- 172.21.15.79:0/3429620160 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068199160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe0580090b0 tx=0x7fe0580046c0 comp rx=0 tx=0).stop 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.604+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 shutdown_connections 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.604+0000 7fe06ea24640 1 --2- 172.21.15.79:0/3429620160 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe068105c60 0x7fe068199160 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.604+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 >> 172.21.15.79:0/3429620160 conn(0x7fe0681011c0 msgr2=0x7fe06818ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.604+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 shutdown_connections 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:32.604+0000 7fe06ea24640 1 -- 172.21.15.79:0/3429620160 wait complete. 2024-06-01T00:45:33.207 INFO:teuthology.orchestra.run.smithi079.stdout:mon is available 2024-06-01T00:45:33.208 INFO:teuthology.orchestra.run.smithi079.stdout:Assimilating anything we can from ceph.conf... 2024-06-01T00:45:33.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:33 smithi079 ceph-mon[28986]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-06-01T00:45:33.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:33 smithi079 ceph-mon[28986]: monmap e1: 1 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-01T00:45:33.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:33 smithi079 ceph-mon[28986]: fsmap 2024-06-01T00:45:33.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:33 smithi079 ceph-mon[28986]: osdmap e1: 0 total, 0 up, 0 in 2024-06-01T00:45:33.323 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:33 smithi079 ceph-mon[28986]: mgrmap e1: no daemons active 2024-06-01T00:45:33.324 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:33 smithi079 ceph-mon[28986]: from='client.? 172.21.15.79:0/3429620160' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-01T00:45:34.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:34.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-06-01T00:45:34.192 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-01T00:45:34.193 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.614+0000 7fdf63fbc640 1 Processor -- start 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.614+0000 7fdf63fbc640 1 -- start start 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.615+0000 7fdf63fbc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.615+0000 7fdf63fbc640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c105500 con 0x7fdf5c104b10 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.615+0000 7fdf61d58640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.615+0000 7fdf61d58640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46586/0 (socket says 172.21.15.79:46586) 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.615+0000 7fdf61d58640 1 -- 172.21.15.79:0/3024505297 learned_addr learned my addr 172.21.15.79:0/3024505297 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.615+0000 7fdf61d58640 1 -- 172.21.15.79:0/3024505297 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c105d10 con 0x7fdf5c104b10 2024-06-01T00:45:34.194 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.616+0000 7fdf61d58640 1 --2- 172.21.15.79:0/3024505297 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c104f30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009e80 tx=0x7fdf4c02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b49a9c51532f11 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.616+0000 7fdf60d56640 1 -- 172.21.15.79:0/3024505297 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf4c031400 con 0x7fdf5c104b10 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.616+0000 7fdf60d56640 1 -- 172.21.15.79:0/3024505297 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c02d050 con 0x7fdf5c104b10 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.616+0000 7fdf60d56640 1 -- 172.21.15.79:0/3024505297 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf4c0316d0 con 0x7fdf5c104b10 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.616+0000 7fdf63fbc640 1 -- 172.21.15.79:0/3024505297 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 msgr2=0x7fdf5c104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.616+0000 7fdf63fbc640 1 --2- 172.21.15.79:0/3024505297 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c104f30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009e80 tx=0x7fdf4c02a7a0 comp rx=0 tx=0).stop 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 -- 172.21.15.79:0/3024505297 shutdown_connections 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 --2- 172.21.15.79:0/3024505297 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c104f30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 -- 172.21.15.79:0/3024505297 >> 172.21.15.79:0/3024505297 conn(0x7fdf5c1000d0 msgr2=0x7fdf5c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 -- 172.21.15.79:0/3024505297 shutdown_connections 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 -- 172.21.15.79:0/3024505297 wait complete. 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 Processor -- start 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 -- start start 2024-06-01T00:45:34.195 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.617+0000 7fdf63fbc640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c079ed0 con 0x7fdf5c104b10 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf61d58640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf61d58640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c079990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46598/0 (socket says 172.21.15.79:46598) 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf61d58640 1 -- 172.21.15.79:0/2332574904 learned_addr learned my addr 172.21.15.79:0/2332574904 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf61d58640 1 -- 172.21.15.79:0/2332574904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c009b30 con 0x7fdf5c104b10 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf61d58640 1 --2- 172.21.15.79:0/2332574904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c079990 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c02acc0 tx=0x7fdf4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf4c03b7d0 con 0x7fdf5c104b10 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c03b930 con 0x7fdf5c104b10 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c19be60 con 0x7fdf5c104b10 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf4c03bc30 con 0x7fdf5c104b10 2024-06-01T00:45:34.196 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c19c2a0 con 0x7fdf5c104b10 2024-06-01T00:45:34.197 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fdf4c038040 con 0x7fdf5c104b10 2024-06-01T00:45:34.197 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.618+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdf4c03a3d0 con 0x7fdf5c104b10 2024-06-01T00:45:34.197 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.619+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c191d40 con 0x7fdf5c104b10 2024-06-01T00:45:34.197 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.621+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fdf4c04b060 con 0x7fdf5c104b10 2024-06-01T00:45:34.198 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.680+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdf5c1062e0 con 0x7fdf5c104b10 2024-06-01T00:45:34.198 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.684+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdf4c03aa60 con 0x7fdf5c104b10 2024-06-01T00:45:34.198 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.684+0000 7fdf5affd640 1 -- 172.21.15.79:0/2332574904 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fdf4c042070 con 0x7fdf5c104b10 2024-06-01T00:45:34.198 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.685+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 msgr2=0x7fdf5c079990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:34.199 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.685+0000 7fdf63fbc640 1 --2- 172.21.15.79:0/2332574904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c079990 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c02acc0 tx=0x7fdf4c004a40 comp rx=0 tx=0).stop 2024-06-01T00:45:34.199 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.685+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 shutdown_connections 2024-06-01T00:45:34.199 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.685+0000 7fdf63fbc640 1 --2- 172.21.15.79:0/2332574904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf5c104b10 0x7fdf5c079990 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:34.199 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.685+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 >> 172.21.15.79:0/2332574904 conn(0x7fdf5c1000d0 msgr2=0x7fdf5c18ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:34.199 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.685+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 shutdown_connections 2024-06-01T00:45:34.199 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:33.685+0000 7fdf63fbc640 1 -- 172.21.15.79:0/2332574904 wait complete. 2024-06-01T00:45:34.199 INFO:teuthology.orchestra.run.smithi079.stdout:Generating new minimal ceph.conf... 2024-06-01T00:45:34.301 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:34 smithi079 ceph-mon[28986]: from='client.? 172.21.15.79:0/2332574904' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-01T00:45:34.302 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:34 smithi079 ceph-mon[28986]: from='client.? 172.21.15.79:0/2332574904' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-01T00:45:35.222 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.611+0000 7fc51dffd640 1 Processor -- start 2024-06-01T00:45:35.222 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.612+0000 7fc51dffd640 1 -- start start 2024-06-01T00:45:35.222 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.612+0000 7fc51dffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518108b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:35.222 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.612+0000 7fc51dffd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518079e00 con 0x7fc518106710 2024-06-01T00:45:35.222 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.612+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518108b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.612+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518108b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46610/0 (socket says 172.21.15.79:46610) 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.612+0000 7fc5177fe640 1 -- 172.21.15.79:0/4217197995 learned_addr learned my addr 172.21.15.79:0/4217197995 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.612+0000 7fc5177fe640 1 -- 172.21.15.79:0/4217197995 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518109040 con 0x7fc518106710 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.613+0000 7fc5177fe640 1 --2- 172.21.15.79:0/4217197995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518108b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc508009820 tx=0x7fc50802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88366cf93981423d server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.613+0000 7fc5167fc640 1 -- 172.21.15.79:0/4217197995 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc50802ad40 con 0x7fc518106710 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.613+0000 7fc5167fc640 1 -- 172.21.15.79:0/4217197995 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc5080328c0 con 0x7fc518106710 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.613+0000 7fc5167fc640 1 -- 172.21.15.79:0/4217197995 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5080307f0 con 0x7fc518106710 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 -- 172.21.15.79:0/4217197995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 msgr2=0x7fc518108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 --2- 172.21.15.79:0/4217197995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518108b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc508009820 tx=0x7fc50802a270 comp rx=0 tx=0).stop 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 -- 172.21.15.79:0/4217197995 shutdown_connections 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 --2- 172.21.15.79:0/4217197995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518108b00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 -- 172.21.15.79:0/4217197995 >> 172.21.15.79:0/4217197995 conn(0x7fc5181000e0 msgr2=0x7fc518102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:35.223 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 -- 172.21.15.79:0/4217197995 shutdown_connections 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 -- 172.21.15.79:0/4217197995 wait complete. 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 Processor -- start 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.614+0000 7fc51dffd640 1 -- start start 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc51dffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc51dffd640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc518199d80 con 0x7fc518106710 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc5177fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46614/0 (socket says 172.21.15.79:46614) 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc5177fe640 1 -- 172.21.15.79:0/1610187192 learned_addr learned my addr 172.21.15.79:0/1610187192 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc5177fe640 1 -- 172.21.15.79:0/1610187192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080094d0 con 0x7fc518106710 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc5177fe640 1 --2- 172.21.15.79:0/1610187192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518199840 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc508030d30 tx=0x7fc508030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc514ff9640 1 -- 172.21.15.79:0/1610187192 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc508032c40 con 0x7fc518106710 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc518199f80 con 0x7fc518106710 2024-06-01T00:45:35.224 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc514ff9640 1 -- 172.21.15.79:0/1610187192 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc508032da0 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc514ff9640 1 -- 172.21.15.79:0/1610187192 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc50803c710 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.615+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51819a3a0 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.616+0000 7fc514ff9640 1 -- 172.21.15.79:0/1610187192 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc50803cc80 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.616+0000 7fc514ff9640 1 -- 172.21.15.79:0/1610187192 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc508046450 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.616+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc518109d60 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.618+0000 7fc514ff9640 1 -- 172.21.15.79:0/1610187192 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fc508037070 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.678+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc51819cf60 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.679+0000 7fc514ff9640 1 -- 172.21.15.79:0/1610187192 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fc50803d480 con 0x7fc518106710 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.680+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 msgr2=0x7fc518199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.680+0000 7fc51dffd640 1 --2- 172.21.15.79:0/1610187192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518199840 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc508030d30 tx=0x7fc508030d60 comp rx=0 tx=0).stop 2024-06-01T00:45:35.225 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.680+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 shutdown_connections 2024-06-01T00:45:35.226 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.680+0000 7fc51dffd640 1 --2- 172.21.15.79:0/1610187192 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc518106710 0x7fc518199840 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:35.226 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.680+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 >> 172.21.15.79:0/1610187192 conn(0x7fc5181000e0 msgr2=0x7fc518100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:35.226 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.680+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 shutdown_connections 2024-06-01T00:45:35.226 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:34.680+0000 7fc51dffd640 1 -- 172.21.15.79:0/1610187192 wait complete. 2024-06-01T00:45:35.226 INFO:teuthology.orchestra.run.smithi079.stdout:Restarting the monitor... 2024-06-01T00:45:35.244 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:35 smithi079 ceph-mon[28986]: from='client.? 172.21.15.79:0/1610187192' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:45:35.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:35 smithi079 systemd[1]: Stopping Ceph mon.smithi079 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:45:35.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:35 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079[28982]: 2024-06-01T00:45:35.466+0000 7fef2acb1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi079 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:45:35.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:35 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079[28982]: 2024-06-01T00:45:35.466+0000 7fef2acb1640 -1 mon.smithi079@0(leader) e1 *** Got Signal Terminated *** 2024-06-01T00:45:36.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:35 smithi079 podman[29228]: 2024-06-01 00:45:35.68259954 +0000 UTC m=+0.294568926 container died b5b85d688ba4d29b523878aa47a018a13b92847da74e362760af515518e5d864 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_CLEAN=True, ceph=True) 2024-06-01T00:45:36.001 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:36 smithi079 podman[29228]: 2024-06-01 00:45:36.000150825 +0000 UTC m=+0.612120211 container cleanup b5b85d688ba4d29b523878aa47a018a13b92847da74e362760af515518e5d864 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-06-01T00:45:36.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:36 smithi079 bash[29228]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079 2024-06-01T00:45:38.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:37 smithi079 podman[29241]: 2024-06-01 00:45:37.971086491 +0000 UTC m=+2.284472276 container remove b5b85d688ba4d29b523878aa47a018a13b92847da74e362760af515518e5d864 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-71c5cfc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-01T00:45:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:38 smithi079 systemd[1]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079.service: Deactivated successfully. 2024-06-01T00:45:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:38 smithi079 systemd[1]: Stopped Ceph mon.smithi079 for 22870f5e-1fb0-11ef-bc9b-c7b262605968. 2024-06-01T00:45:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:38 smithi079 systemd[1]: Starting Ceph mon.smithi079 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:45:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:38 smithi079 podman[29336]: 2024-06-01 00:45:38.346609575 +0000 UTC m=+0.016765728 image pull 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe quay.ceph.io/ceph-ci/ceph:reef 2024-06-01T00:45:39.205 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:38 smithi079 podman[29336]: 2024-06-01 00:45:38.927456668 +0000 UTC m=+0.597612799 container create 9cf2014362517e1e6223e305a09209c1cd79feaa8b32c8f5b5bb0f42500b2f5e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b) 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 podman[29336]: 2024-06-01 00:45:39.129551138 +0000 UTC m=+0.799707279 container init 9cf2014362517e1e6223e305a09209c1cd79feaa8b32c8f5b5bb0f42500b2f5e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528) 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 podman[29336]: 2024-06-01 00:45:39.132800089 +0000 UTC m=+0.802956228 container start 9cf2014362517e1e6223e305a09209c1cd79feaa8b32c8f5b5bb0f42500b2f5e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, CEPH_POINT_RELEASE=, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: set uid:gid to 167:167 (ceph:ceph) 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: ceph version 18.2.2-1387-g71c5cfc9 (71c5cfc99f93c162032256d34d9b60b8288fad91) reef (stable), process ceph-mon, pid 2 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: pidfile_write: ignore empty --pid-file 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: load: jerasure load: lrc 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: RocksDB version: 7.9.2 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Git sha 0 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Compile date 2024-05-31 16:05:45 2024-06-01T00:45:39.206 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: DB SUMMARY 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: DB Session ID: RO0QHSZ31IIQN3O2DOTL 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: CURRENT file: CURRENT 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: IDENTITY file: IDENTITY 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi079/store.db dir, Total Num: 1, files: 000008.sst 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi079/store.db: 000009.log size: 98489 ; 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.error_if_exists: 0 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.create_if_missing: 0 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.paranoid_checks: 1 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-01T00:45:39.207 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.env: 0x55a61e163ba0 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.fs: PosixFileSystem 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.info_log: 0x55a61f470dc0 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_file_opening_threads: 16 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.statistics: (nil) 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.use_fsync: 0 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_log_file_size: 0 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.keep_log_file_num: 1000 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.recycle_log_file_num: 0 2024-06-01T00:45:39.208 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.allow_fallocate: 1 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.allow_mmap_reads: 0 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.allow_mmap_writes: 0 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.use_direct_reads: 0 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.create_missing_column_families: 0 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.db_log_dir: 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.wal_dir: 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-01T00:45:39.209 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.advise_random_on_open: 1 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.db_write_buffer_size: 0 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.write_buffer_manager: 0x55a61f442140 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.rate_limiter: (nil) 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.wal_recovery_mode: 2 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.enable_thread_tracking: 0 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.enable_pipelined_write: 0 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.unordered_write: 0 2024-06-01T00:45:39.210 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.row_cache: None 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.wal_filter: None 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.allow_ingest_behind: 0 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.two_write_queues: 0 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.manual_wal_flush: 0 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.wal_compression: 0 2024-06-01T00:45:39.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.atomic_flush: 0 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.log_readahead_size: 0 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.best_efforts_recovery: 0 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.allow_data_in_errors: 0 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.db_host_id: __hostname__ 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-01T00:45:39.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_background_jobs: 2 2024-06-01T00:45:39.214 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_background_compactions: -1 2024-06-01T00:45:39.214 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_subcompactions: 1 2024-06-01T00:45:39.214 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-01T00:45:39.214 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-01T00:45:39.214 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-01T00:45:39.215 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_total_wal_size: 0 2024-06-01T00:45:39.215 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-01T00:45:39.215 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-01T00:45:39.215 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-01T00:45:39.215 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_open_files: -1 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bytes_per_sync: 0 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_readahead_size: 0 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_background_flushes: -1 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Compression algorithms supported: 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kZSTD supported: 0 2024-06-01T00:45:39.216 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kXpressCompression supported: 0 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kBZip2Compression supported: 0 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kLZ4Compression supported: 1 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kZlibCompression supported: 1 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kLZ4HCCompression supported: 1 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: kSnappyCompression supported: 1 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi079/store.db/MANIFEST-000010 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.merge_operator: 2024-06-01T00:45:39.217 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_filter: None 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_filter_factory: None 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.sst_partitioner_factory: None 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a61f470a60) 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_type: 0 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: data_block_index_type: 0 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_shortening: 1 2024-06-01T00:45:39.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: checksum: 4 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: no_block_cache: 0 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache: 0x55a61f467350 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_options: 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: capacity : 536870912 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: num_shard_bits : 4 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: strict_capacity_limit : 0 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_compressed: (nil) 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: persistent_cache: (nil) 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_size: 4096 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_size_deviation: 10 2024-06-01T00:45:39.219 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_restart_interval: 16 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_block_restart_interval: 1 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: metadata_block_size: 4096 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: partition_filters: 0 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: use_delta_encoding: 1 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: filter_policy: bloomfilter 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: whole_key_filtering: 1 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: verify_compression: 0 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: format_version: 5 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: enable_index_compression: 1 2024-06-01T00:45:39.220 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_align: 0 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: max_auto_readahead_size: 262144 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: prepopulate_block_cache: 0 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: initial_auto_readahead_size: 8192 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.write_buffer_size: 33554432 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_write_buffer_number: 2 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression: NoCompression 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression: Disabled 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.prefix_extractor: nullptr 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-01T00:45:39.221 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.num_levels: 7 2024-06-01T00:45:39.222 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-01T00:45:39.222 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-01T00:45:39.222 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-01T00:45:39.222 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-01T00:45:39.222 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-01T00:45:39.222 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-01T00:45:39.222 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.level: 32767 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.strategy: 0 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-01T00:45:39.223 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.enabled: false 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.target_file_size_base: 67108864 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-01T00:45:39.224 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-01T00:45:39.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.arena_block_size: 1048576 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.disable_auto_compactions: 0 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-01T00:45:39.226 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.table_properties_collectors: 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.inplace_update_support: 0 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-01T00:45:39.227 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.bloom_locality: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.max_successive_merges: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.paranoid_file_checks: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.force_consistency_checks: 1 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.report_bg_io_stats: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.ttl: 2592000 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.enable_blob_files: false 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.min_blob_size: 0 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.blob_file_size: 268435456 2024-06-01T00:45:39.228 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.blob_file_starting_level: 0 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi079/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 80d0d86f-b33b-43f9-98c9-c7cc52d5a481 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717202739160071, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-06-01T00:45:39.229 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717202739161068, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93766, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 91800, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84364, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717202739, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "80d0d86f-b33b-43f9-98c9-c7cc52d5a481", "db_session_id": "RO0QHSZ31IIQN3O2DOTL", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-06-01T00:45:39.230 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717202739161168, "job": 1, "event": "recovery_finished"} 2024-06-01T00:45:39.230 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-06-01T00:45:39.230 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi079/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a61f490e00 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: DB pointer 0x55a61f584000 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** DB Stats ** 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T00:45:39.231 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: L0 2/0 93.44 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Sum 2/0 93.44 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-01T00:45:39.232 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative compaction: 0.00 GB write, 27.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T00:45:39.233 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval compaction: 0.00 GB write, 27.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Block cache BinnedLRUCache@0x55a61f467350#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: starting mon.smithi079 rank 0 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi079 fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???) e1 preinit fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???).mds e1 new map 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???).mds e1 print_map 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: e1 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-01T00:45:39.234 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-01T00:45:39.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: legacy client fscid: -1 2024-06-01T00:45:39.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-06-01T00:45:39.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: No filesystems configured 2024-06-01T00:45:39.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-01T00:45:39.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T00:45:39.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T00:45:39.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T00:45:39.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-01T00:45:39.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 bash[29336]: 9cf2014362517e1e6223e305a09209c1cd79feaa8b32c8f5b5bb0f42500b2f5e 2024-06-01T00:45:39.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 systemd[1]: Started Ceph mon.smithi079 for 22870f5e-1fb0-11ef-bc9b-c7b262605968. 2024-06-01T00:45:39.236 INFO:teuthology.orchestra.run.smithi079.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-01T00:45:39.474 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-06-01T00:45:39.475 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: monmap e1: 1 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-01T00:45:39.475 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: fsmap 2024-06-01T00:45:39.475 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: osdmap e1: 0 total, 0 up, 0 in 2024-06-01T00:45:39.475 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:39 smithi079 ceph-mon[29356]: mgrmap e1: no daemons active 2024-06-01T00:45:40.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.655+0000 7fc0b211e640 1 Processor -- start 2024-06-01T00:45:40.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.655+0000 7fc0b211e640 1 -- start start 2024-06-01T00:45:40.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.655+0000 7fc0b211e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:40.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.655+0000 7fc0b211e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac106630 con 0x7fc0ac105c40 2024-06-01T00:45:40.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.655+0000 7fc0ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:40.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.655+0000 7fc0ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46624/0 (socket says 172.21.15.79:46624) 2024-06-01T00:45:40.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.655+0000 7fc0ab7fe640 1 -- 172.21.15.79:0/2195790144 learned_addr learned my addr 172.21.15.79:0/2195790144 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:40.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.656+0000 7fc0ab7fe640 1 -- 172.21.15.79:0/2195790144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac106e40 con 0x7fc0ac105c40 2024-06-01T00:45:40.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.657+0000 7fc0ab7fe640 1 --2- 172.21.15.79:0/2195790144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac106060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009820 tx=0x7fc09c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f10554cb69db0f server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:40.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0aa7fc640 1 -- 172.21.15.79:0/2195790144 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09c02ad40 con 0x7fc0ac105c40 2024-06-01T00:45:40.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0aa7fc640 1 -- 172.21.15.79:0/2195790144 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc09c0328c0 con 0x7fc0ac105c40 2024-06-01T00:45:40.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0aa7fc640 1 -- 172.21.15.79:0/2195790144 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09c0307f0 con 0x7fc0ac105c40 2024-06-01T00:45:40.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0b211e640 1 -- 172.21.15.79:0/2195790144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 msgr2=0x7fc0ac106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:40.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0b211e640 1 --2- 172.21.15.79:0/2195790144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac106060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009820 tx=0x7fc09c02a270 comp rx=0 tx=0).stop 2024-06-01T00:45:40.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0b211e640 1 -- 172.21.15.79:0/2195790144 shutdown_connections 2024-06-01T00:45:40.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0b211e640 1 --2- 172.21.15.79:0/2195790144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac106060 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:40.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0b211e640 1 -- 172.21.15.79:0/2195790144 >> 172.21.15.79:0/2195790144 conn(0x7fc0ac1011c0 msgr2=0x7fc0ac103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:40.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0b211e640 1 -- 172.21.15.79:0/2195790144 shutdown_connections 2024-06-01T00:45:40.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.658+0000 7fc0b211e640 1 -- 172.21.15.79:0/2195790144 wait complete. 2024-06-01T00:45:40.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.659+0000 7fc0b211e640 1 Processor -- start 2024-06-01T00:45:40.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.659+0000 7fc0b211e640 1 -- start start 2024-06-01T00:45:40.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.659+0000 7fc0b211e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:40.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.659+0000 7fc0b211e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac199d90 con 0x7fc0ac105c40 2024-06-01T00:45:40.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.659+0000 7fc0ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:40.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.659+0000 7fc0ab7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46626/0 (socket says 172.21.15.79:46626) 2024-06-01T00:45:40.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.659+0000 7fc0ab7fe640 1 -- 172.21.15.79:0/1495006125 learned_addr learned my addr 172.21.15.79:0/1495006125 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:40.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0ab7fe640 1 -- 172.21.15.79:0/1495006125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09c0094d0 con 0x7fc0ac105c40 2024-06-01T00:45:40.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0ab7fe640 1 --2- 172.21.15.79:0/1495006125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac199850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc09c030d30 tx=0x7fc09c030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:40.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0a8ff9640 1 -- 172.21.15.79:0/1495006125 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09c032c40 con 0x7fc0ac105c40 2024-06-01T00:45:40.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0a8ff9640 1 -- 172.21.15.79:0/1495006125 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc09c032da0 con 0x7fc0ac105c40 2024-06-01T00:45:40.304 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac199f90 con 0x7fc0ac105c40 2024-06-01T00:45:40.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0a8ff9640 1 -- 172.21.15.79:0/1495006125 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc09c03c6a0 con 0x7fc0ac105c40 2024-06-01T00:45:40.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ac19a3b0 con 0x7fc0ac105c40 2024-06-01T00:45:40.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0a8ff9640 1 -- 172.21.15.79:0/1495006125 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc09c03cbe0 con 0x7fc0ac105c40 2024-06-01T00:45:40.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.660+0000 7fc0a8ff9640 1 -- 172.21.15.79:0/1495006125 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc09c0463c0 con 0x7fc0ac105c40 2024-06-01T00:45:40.305 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.661+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ac10a070 con 0x7fc0ac105c40 2024-06-01T00:45:40.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.663+0000 7fc0a8ff9640 1 -- 172.21.15.79:0/1495006125 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fc09c037070 con 0x7fc0ac105c40 2024-06-01T00:45:40.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.722+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc0ac19ccc0 con 0x7fc0ac105c40 2024-06-01T00:45:40.306 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.727+0000 7fc0a8ff9640 1 -- 172.21.15.79:0/1495006125 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc09c03da90 con 0x7fc0ac105c40 2024-06-01T00:45:40.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.728+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 msgr2=0x7fc0ac199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:40.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.728+0000 7fc0b211e640 1 --2- 172.21.15.79:0/1495006125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac199850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc09c030d30 tx=0x7fc09c030d60 comp rx=0 tx=0).stop 2024-06-01T00:45:40.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.728+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 shutdown_connections 2024-06-01T00:45:40.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.728+0000 7fc0b211e640 1 --2- 172.21.15.79:0/1495006125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0ac105c40 0x7fc0ac199850 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:40.307 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.728+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 >> 172.21.15.79:0/1495006125 conn(0x7fc0ac1011c0 msgr2=0x7fc0ac101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:40.308 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.728+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 shutdown_connections 2024-06-01T00:45:40.308 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:39.728+0000 7fc0b211e640 1 -- 172.21.15.79:0/1495006125 wait complete. 2024-06-01T00:45:40.308 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-01T00:45:40.320 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-01T00:45:40.320 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mgr... 2024-06-01T00:45:40.321 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:9283 ... 2024-06-01T00:45:40.321 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8765 ... 2024-06-01T00:45:40.322 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8443 ... 2024-06-01T00:45:40.563 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mgr.smithi079.emylav 2024-06-01T00:45:40.563 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mgr.smithi079.emylav.service: Unit ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mgr.smithi079.emylav.service not loaded. 2024-06-01T00:45:40.705 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968.target.wants/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mgr.smithi079.emylav.service → /etc/systemd/system/ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@.service. 2024-06-01T00:45:40.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:40 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1495006125' entity='client.admin' 2024-06-01T00:45:41.271 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-01T00:45:41.271 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-06-01T00:45:41.278 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-01T00:45:41.279 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-06-01T00:45:41.279 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-06-01T00:45:41.279 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-06-01T00:45:41.287 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-01T00:45:41.287 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-06-01T00:45:41.295 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-01T00:45:41.295 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-06-01T00:45:41.295 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-06-01T00:45:41.295 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-06-01T00:45:41.295 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr to start... 2024-06-01T00:45:41.295 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr... 2024-06-01T00:45:42.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:41 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2978453568' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-01T00:45:42.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:42.895 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "22870f5e-1fb0-11ef-bc9b-c7b262605968", 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-01T00:45:42.896 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-01T00:45:42.897 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:42.898 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-01T00:45:42.898 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-01T00:45:42.898 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-01T00:45:42.907 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-01T00:45:42.907 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-01T00:45:42.907 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-01T00:45:42.908 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-01T00:45:42.909 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-06-01T00:45:32.073287+0000", 2024-06-01T00:45:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.713+0000 7fea41169640 1 Processor -- start 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.713+0000 7fea41169640 1 -- start start 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.713+0000 7fea41169640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c108ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.713+0000 7fea41169640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c079de0 con 0x7fea3c1066e0 2024-06-01T00:45:42.911 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.713+0000 7fea3ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c108ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.713+0000 7fea3ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c108ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46650/0 (socket says 172.21.15.79:46650) 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.713+0000 7fea3ad9d640 1 -- 172.21.15.79:0/3247795 learned_addr learned my addr 172.21.15.79:0/3247795 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.714+0000 7fea3ad9d640 1 -- 172.21.15.79:0/3247795 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c109010 con 0x7fea3c1066e0 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.714+0000 7fea3ad9d640 1 --2- 172.21.15.79:0/3247795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c108ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009820 tx=0x7fea2c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d86ba5b9a51703 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea39d9b640 1 -- 172.21.15.79:0/3247795 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea2c02ad40 con 0x7fea3c1066e0 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea39d9b640 1 -- 172.21.15.79:0/3247795 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fea2c0328c0 con 0x7fea3c1066e0 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea39d9b640 1 -- 172.21.15.79:0/3247795 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea2c0307f0 con 0x7fea3c1066e0 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea41169640 1 -- 172.21.15.79:0/3247795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 msgr2=0x7fea3c108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea41169640 1 --2- 172.21.15.79:0/3247795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c108ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009820 tx=0x7fea2c02a270 comp rx=0 tx=0).stop 2024-06-01T00:45:42.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea41169640 1 -- 172.21.15.79:0/3247795 shutdown_connections 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea41169640 1 --2- 172.21.15.79:0/3247795 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c108ad0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea41169640 1 -- 172.21.15.79:0/3247795 >> 172.21.15.79:0/3247795 conn(0x7fea3c1000d0 msgr2=0x7fea3c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea41169640 1 -- 172.21.15.79:0/3247795 shutdown_connections 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.715+0000 7fea41169640 1 -- 172.21.15.79:0/3247795 wait complete. 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.716+0000 7fea41169640 1 Processor -- start 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.716+0000 7fea41169640 1 -- start start 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.716+0000 7fea41169640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c19ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.716+0000 7fea41169640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c19bf80 con 0x7fea3c1066e0 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea3ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c19ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea3ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c19ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46654/0 (socket says 172.21.15.79:46654) 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea3ad9d640 1 -- 172.21.15.79:0/2978453568 learned_addr learned my addr 172.21.15.79:0/2978453568 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:42.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea3ad9d640 1 -- 172.21.15.79:0/2978453568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c0094d0 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea3ad9d640 1 --2- 172.21.15.79:0/2978453568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c19ba40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009940 tx=0x7fea2c032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea23fff640 1 -- 172.21.15.79:0/2978453568 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea2c032df0 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c19c180 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea23fff640 1 -- 172.21.15.79:0/2978453568 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fea2c030cc0 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea23fff640 1 -- 172.21.15.79:0/2978453568 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea2c03c6b0 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c19c5a0 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea23fff640 1 -- 172.21.15.79:0/2978453568 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fea2c03cc70 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.717+0000 7fea23fff640 1 -- 172.21.15.79:0/2978453568 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fea2c0463c0 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.718+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c109d30 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.720+0000 7fea23fff640 1 -- 172.21.15.79:0/2978453568 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7fea2c037070 con 0x7fea3c1066e0 2024-06-01T00:45:42.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.784+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fea3c109f20 con 0x7fea3c1066e0 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.784+0000 7fea23fff640 1 -- 172.21.15.79:0/2978453568 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fea2c030e30 con 0x7fea3c1066e0 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.785+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 msgr2=0x7fea3c19ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.785+0000 7fea41169640 1 --2- 172.21.15.79:0/2978453568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c19ba40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009940 tx=0x7fea2c032c60 comp rx=0 tx=0).stop 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.785+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 shutdown_connections 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.785+0000 7fea41169640 1 --2- 172.21.15.79:0/2978453568 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fea3c1066e0 0x7fea3c19ba40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.785+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 >> 172.21.15.79:0/2978453568 conn(0x7fea3c1000d0 msgr2=0x7fea3c100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.786+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 shutdown_connections 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:41.786+0000 7fea41169640 1 -- 172.21.15.79:0/2978453568 wait complete. 2024-06-01T00:45:42.915 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (1/15)... 2024-06-01T00:45:45.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:45 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1703676636' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-01T00:45:45.982 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:45.994 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "22870f5e-1fb0-11ef-bc9b-c7b262605968", 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-06-01T00:45:45.995 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-01T00:45:45.996 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-01T00:45:45.997 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-01T00:45:45.997 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-01T00:45:45.997 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:45.997 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-01T00:45:45.997 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-01T00:45:45.997 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-01T00:45:45.998 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-01T00:45:45.998 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-01T00:45:45.998 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-01T00:45:45.998 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-01T00:45:45.998 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-01T00:45:45.999 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-01T00:45:45.999 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:45.999 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-01T00:45:45.999 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:45.999 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-01T00:45:45.999 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-01T00:45:46.000 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:46.000 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-01T00:45:46.000 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-01T00:45:46.000 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-01T00:45:46.000 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-01T00:45:46.000 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-01T00:45:46.000 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-01T00:45:46.001 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-06-01T00:45:46.001 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:46.001 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-01T00:45:46.001 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:46.001 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-01T00:45:46.001 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:46.001 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-06-01T00:45:32.073287+0000", 2024-06-01T00:45:46.002 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-01T00:45:46.002 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:46.002 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-01T00:45:46.002 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-06-01T00:45:46.002 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.303+0000 7f9fbaa6c640 1 Processor -- start 2024-06-01T00:45:46.002 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.303+0000 7f9fbaa6c640 1 -- start start 2024-06-01T00:45:46.002 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.303+0000 7f9fbaa6c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb4104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:46.003 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.303+0000 7f9fbaa6c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb4105500 con 0x7f9fb4104b10 2024-06-01T00:45:46.003 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.304+0000 7f9fb8808640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb4104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:46.003 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.304+0000 7f9fb8808640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb4104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56236/0 (socket says 172.21.15.79:56236) 2024-06-01T00:45:46.003 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.304+0000 7f9fb8808640 1 -- 172.21.15.79:0/502002387 learned_addr learned my addr 172.21.15.79:0/502002387 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:46.003 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.304+0000 7f9fb8808640 1 -- 172.21.15.79:0/502002387 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4105d10 con 0x7f9fb4104b10 2024-06-01T00:45:46.003 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.305+0000 7f9fb8808640 1 --2- 172.21.15.79:0/502002387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb4104f30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009820 tx=0x7f9fa802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02b6ffb66df7738 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:46.004 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.305+0000 7f9fb37fe640 1 -- 172.21.15.79:0/502002387 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa802ad40 con 0x7f9fb4104b10 2024-06-01T00:45:46.004 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.305+0000 7f9fb37fe640 1 -- 172.21.15.79:0/502002387 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9fa80328c0 con 0x7f9fb4104b10 2024-06-01T00:45:46.004 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.305+0000 7f9fb37fe640 1 -- 172.21.15.79:0/502002387 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa80307f0 con 0x7f9fb4104b10 2024-06-01T00:45:46.004 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/502002387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 msgr2=0x7f9fb4104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:46.004 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 --2- 172.21.15.79:0/502002387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb4104f30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009820 tx=0x7f9fa802a270 comp rx=0 tx=0).stop 2024-06-01T00:45:46.004 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/502002387 shutdown_connections 2024-06-01T00:45:46.004 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 --2- 172.21.15.79:0/502002387 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb4104f30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/502002387 >> 172.21.15.79:0/502002387 conn(0x7f9fb41000d0 msgr2=0x7f9fb41024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/502002387 shutdown_connections 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/502002387 wait complete. 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 Processor -- start 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.306+0000 7f9fbaa6c640 1 -- start start 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fbaa6c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb41998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fbaa6c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb4199de0 con 0x7f9fb4104b10 2024-06-01T00:45:46.005 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb8808640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb41998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:46.006 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb8808640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb41998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56242/0 (socket says 172.21.15.79:56242) 2024-06-01T00:45:46.006 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb8808640 1 -- 172.21.15.79:0/1703676636 learned_addr learned my addr 172.21.15.79:0/1703676636 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:46.006 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb8808640 1 -- 172.21.15.79:0/1703676636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80094d0 con 0x7f9fb4104b10 2024-06-01T00:45:46.006 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb8808640 1 --2- 172.21.15.79:0/1703676636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb41998a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009940 tx=0x7f9fa8032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:46.006 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb1ffb640 1 -- 172.21.15.79:0/1703676636 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa8041070 con 0x7f9fb4104b10 2024-06-01T00:45:46.006 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb1ffb640 1 -- 172.21.15.79:0/1703676636 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9fa8030cc0 con 0x7f9fb4104b10 2024-06-01T00:45:46.007 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb4199fe0 con 0x7f9fb4104b10 2024-06-01T00:45:46.007 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.307+0000 7f9fb1ffb640 1 -- 172.21.15.79:0/1703676636 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa803c690 con 0x7f9fb4104b10 2024-06-01T00:45:46.007 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.308+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb419a400 con 0x7f9fb4104b10 2024-06-01T00:45:46.007 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.308+0000 7f9fb1ffb640 1 -- 172.21.15.79:0/1703676636 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f9fa803cb80 con 0x7f9fb4104b10 2024-06-01T00:45:46.007 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.308+0000 7f9fb1ffb640 1 -- 172.21.15.79:0/1703676636 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9fa803a3b0 con 0x7f9fb4104b10 2024-06-01T00:45:46.007 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.309+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb4061ae0 con 0x7f9fb4104b10 2024-06-01T00:45:46.008 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.311+0000 7f9fb1ffb640 1 -- 172.21.15.79:0/1703676636 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (secure 0 0 0) 0x7f9fa8046e60 con 0x7f9fb4104b10 2024-06-01T00:45:46.008 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.374+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9fb4193480 con 0x7f9fb4104b10 2024-06-01T00:45:46.008 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.374+0000 7f9fb1ffb640 1 -- 172.21.15.79:0/1703676636 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f9fa8037070 con 0x7f9fb4104b10 2024-06-01T00:45:46.008 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.375+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 msgr2=0x7f9fb41998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:46.008 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.375+0000 7f9fbaa6c640 1 --2- 172.21.15.79:0/1703676636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb41998a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009940 tx=0x7f9fa8032c60 comp rx=0 tx=0).stop 2024-06-01T00:45:46.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.375+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 shutdown_connections 2024-06-01T00:45:46.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.375+0000 7f9fbaa6c640 1 --2- 172.21.15.79:0/1703676636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fb4104b10 0x7f9fb41998a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:46.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.375+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 >> 172.21.15.79:0/1703676636 conn(0x7f9fb41000d0 msgr2=0x7f9fb4190650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:46.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.376+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 shutdown_connections 2024-06-01T00:45:46.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:45.376+0000 7f9fbaa6c640 1 -- 172.21.15.79:0/1703676636 wait complete. 2024-06-01T00:45:46.009 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (2/15)... 2024-06-01T00:45:46.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: Activating manager daemon smithi079.emylav 2024-06-01T00:45:46.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: mgrmap e2: smithi079.emylav(active, starting, since 0.00305212s) 2024-06-01T00:45:46.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T00:45:46.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T00:45:46.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T00:45:46.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:45:46.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr metadata", "who": "smithi079.emylav", "id": "smithi079.emylav"}]: dispatch 2024-06-01T00:45:46.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: Manager daemon smithi079.emylav is now available 2024-06-01T00:45:46.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/mirror_snapshot_schedule"}]: dispatch 2024-06-01T00:45:46.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/trash_purge_schedule"}]: dispatch 2024-06-01T00:45:46.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' 2024-06-01T00:45:46.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' 2024-06-01T00:45:46.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:46 smithi079 ceph-mon[29356]: from='mgr.14100 172.21.15.79:0/1895587888' entity='mgr.smithi079.emylav' 2024-06-01T00:45:48.422 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:48 smithi079 ceph-mon[29356]: mgrmap e3: smithi079.emylav(active, since 1.00672s) 2024-06-01T00:45:49.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:49.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-06-01T00:45:49.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "22870f5e-1fb0-11ef-bc9b-c7b262605968", 2024-06-01T00:45:49.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-06-01T00:45:49.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-01T00:45:49.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-01T00:45:49.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-01T00:45:49.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-01T00:45:49.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-01T00:45:49.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:49.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-01T00:45:49.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-01T00:45:49.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-01T00:45:49.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-01T00:45:49.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-01T00:45:49.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-01T00:45:49.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-01T00:45:49.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-01T00:45:49.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-01T00:45:49.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:49.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-01T00:45:49.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:49.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-01T00:45:49.297 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-01T00:45:49.297 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:49.297 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-01T00:45:49.297 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-01T00:45:49.297 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-01T00:45:49.297 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-06-01T00:45:32.073287+0000", 2024-06-01T00:45:49.298 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e6bb4640 1 Processor -- start 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e6bb4640 1 -- start start 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e6bb4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e6bb4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0106630 con 0x7fc0e0105c40 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e4950640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:49.299 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e4950640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56310/0 (socket says 172.21.15.79:56310) 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e4950640 1 -- 172.21.15.79:0/1333485613 learned_addr learned my addr 172.21.15.79:0/1333485613 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.421+0000 7fc0e4950640 1 -- 172.21.15.79:0/1333485613 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e0106e40 con 0x7fc0e0105c40 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.422+0000 7fc0e4950640 1 --2- 172.21.15.79:0/1333485613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0106060 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009820 tx=0x7fc0d402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00761bb4928469 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.422+0000 7fc0df7fe640 1 -- 172.21.15.79:0/1333485613 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0d402ad40 con 0x7fc0e0105c40 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.422+0000 7fc0df7fe640 1 -- 172.21.15.79:0/1333485613 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc0d40328c0 con 0x7fc0e0105c40 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.422+0000 7fc0df7fe640 1 -- 172.21.15.79:0/1333485613 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0d40307f0 con 0x7fc0e0105c40 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.423+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1333485613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 msgr2=0x7fc0e0106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.423+0000 7fc0e6bb4640 1 --2- 172.21.15.79:0/1333485613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0106060 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009820 tx=0x7fc0d402a270 comp rx=0 tx=0).stop 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.423+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1333485613 shutdown_connections 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.423+0000 7fc0e6bb4640 1 --2- 172.21.15.79:0/1333485613 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0106060 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.423+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1333485613 >> 172.21.15.79:0/1333485613 conn(0x7fc0e01011c0 msgr2=0x7fc0e0103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.423+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1333485613 shutdown_connections 2024-06-01T00:45:49.300 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.423+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1333485613 wait complete. 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e6bb4640 1 Processor -- start 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e6bb4640 1 -- start start 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e6bb4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e6bb4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0199db0 con 0x7fc0e0105c40 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e4950640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e4950640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56312/0 (socket says 172.21.15.79:56312) 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e4950640 1 -- 172.21.15.79:0/1545489133 learned_addr learned my addr 172.21.15.79:0/1545489133 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e4950640 1 -- 172.21.15.79:0/1545489133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d40094d0 con 0x7fc0e0105c40 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.424+0000 7fc0e4950640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0199870 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4030d30 tx=0x7fc0d4030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.425+0000 7fc0ddffb640 1 -- 172.21.15.79:0/1545489133 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0d4032c40 con 0x7fc0e0105c40 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.425+0000 7fc0ddffb640 1 -- 172.21.15.79:0/1545489133 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc0d4032da0 con 0x7fc0e0105c40 2024-06-01T00:45:49.301 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.425+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e0199fb0 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.425+0000 7fc0ddffb640 1 -- 172.21.15.79:0/1545489133 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0d403c6a0 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.425+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e019a3d0 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.426+0000 7fc0ddffb640 1 -- 172.21.15.79:0/1545489133 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) v1 ==== 44855+0+0 (secure 0 0 0) 0x7fc0d403c800 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.426+0000 7fc0ddffb640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7fc0d0037860 0x7fc0d0039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.426+0000 7fc0ddffb640 1 -- 172.21.15.79:0/1545489133 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc0d406aa00 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.426+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e0061ae0 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.426+0000 7fc0dffff640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7fc0d0037860 0x7fc0d0039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.426+0000 7fc0dffff640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7fc0d0037860 0x7fc0d0039d20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0098c0 tx=0x7fc0cc006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.431+0000 7fc0ddffb640 1 -- 172.21.15.79:0/1545489133 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc0d4047460 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.663+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc0e019cf70 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.663+0000 7fc0ddffb640 1 -- 172.21.15.79:0/1545489133 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1245 (secure 0 0 0) 0x7fc0d403bb30 con 0x7fc0e0105c40 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7fc0d0037860 msgr2=0x7fc0d0039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:49.302 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7fc0d0037860 0x7fc0d0039d20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0098c0 tx=0x7fc0cc006db0 comp rx=0 tx=0).stop 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 msgr2=0x7fc0e0199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0199870 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4030d30 tx=0x7fc0d4030d60 comp rx=0 tx=0).stop 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 shutdown_connections 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7fc0d0037860 0x7fc0d0039d20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 --2- 172.21.15.79:0/1545489133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc0e0105c40 0x7fc0e0199870 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 >> 172.21.15.79:0/1545489133 conn(0x7fc0e01011c0 msgr2=0x7fc0e0101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 shutdown_connections 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:48.665+0000 7fc0e6bb4640 1 -- 172.21.15.79:0/1545489133 wait complete. 2024-06-01T00:45:49.303 INFO:teuthology.orchestra.run.smithi079.stdout:mgr is available 2024-06-01T00:45:49.400 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:49 smithi079 ceph-mon[29356]: mgrmap e4: smithi079.emylav(active, since 2s) 2024-06-01T00:45:49.400 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:49 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1545489133' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-01T00:45:50.468 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:50 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3633920551' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-01T00:45:50.481 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:50.481 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-06-01T00:45:50.481 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-01T00:45:50.482 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.700+0000 7f7a627c4640 1 Processor -- start 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.700+0000 7f7a627c4640 1 -- start start 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.700+0000 7f7a627c4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.700+0000 7f7a627c4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c106630 con 0x7f7a5c105c40 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.701+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.701+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56328/0 (socket says 172.21.15.79:56328) 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.701+0000 7f7a5bfff640 1 -- 172.21.15.79:0/2594397560 learned_addr learned my addr 172.21.15.79:0/2594397560 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.701+0000 7f7a5bfff640 1 -- 172.21.15.79:0/2594397560 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c106e40 con 0x7f7a5c105c40 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a5bfff640 1 --2- 172.21.15.79:0/2594397560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c106060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009820 tx=0x7f7a4c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1324e6aceab58a4c server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a5b7fe640 1 -- 172.21.15.79:0/2594397560 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c02ad40 con 0x7f7a5c105c40 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a5b7fe640 1 -- 172.21.15.79:0/2594397560 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7a4c0328c0 con 0x7f7a5c105c40 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a5b7fe640 1 -- 172.21.15.79:0/2594397560 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c0307f0 con 0x7f7a5c105c40 2024-06-01T00:45:50.483 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a627c4640 1 -- 172.21.15.79:0/2594397560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 msgr2=0x7f7a5c106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a627c4640 1 --2- 172.21.15.79:0/2594397560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c106060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009820 tx=0x7f7a4c02a270 comp rx=0 tx=0).stop 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a627c4640 1 -- 172.21.15.79:0/2594397560 shutdown_connections 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a627c4640 1 --2- 172.21.15.79:0/2594397560 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c106060 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a627c4640 1 -- 172.21.15.79:0/2594397560 >> 172.21.15.79:0/2594397560 conn(0x7f7a5c1011c0 msgr2=0x7f7a5c103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a627c4640 1 -- 172.21.15.79:0/2594397560 shutdown_connections 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.702+0000 7f7a627c4640 1 -- 172.21.15.79:0/2594397560 wait complete. 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.703+0000 7f7a627c4640 1 Processor -- start 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.703+0000 7f7a627c4640 1 -- start start 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.703+0000 7f7a627c4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.703+0000 7f7a627c4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5c199db0 con 0x7f7a5c105c40 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.703+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.703+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56338/0 (socket says 172.21.15.79:56338) 2024-06-01T00:45:50.484 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.703+0000 7f7a5bfff640 1 -- 172.21.15.79:0/3633920551 learned_addr learned my addr 172.21.15.79:0/3633920551 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.704+0000 7f7a5bfff640 1 -- 172.21.15.79:0/3633920551 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c0094d0 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.704+0000 7f7a5bfff640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c199870 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c030d30 tx=0x7f7a4c030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.704+0000 7f7a59ffb640 1 -- 172.21.15.79:0/3633920551 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c032c40 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.704+0000 7f7a59ffb640 1 -- 172.21.15.79:0/3633920551 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7a4c032da0 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.704+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5c199fb0 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.704+0000 7f7a59ffb640 1 -- 172.21.15.79:0/3633920551 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4c03c6a0 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.704+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5c19a3d0 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.705+0000 7f7a59ffb640 1 -- 172.21.15.79:0/3633920551 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f7a4c03d480 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.705+0000 7f7a59ffb640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f7a48037860 0x7f7a48039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.705+0000 7f7a59ffb640 1 -- 172.21.15.79:0/3633920551 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7a4c06a950 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.705+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5c10a070 con 0x7f7a5c105c40 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.705+0000 7f7a53fff640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f7a48037860 0x7f7a48039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:50.485 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.706+0000 7f7a53fff640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f7a48037860 0x7f7a48039d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a440098c0 tx=0x7f7a44006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.710+0000 7f7a59ffb640 1 -- 172.21.15.79:0/3633920551 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7a4c03cb80 con 0x7f7a5c105c40 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.884+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7a5c19cce0 con 0x7f7a5c105c40 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.885+0000 7f7a59ffb640 1 -- 172.21.15.79:0/3633920551 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f7a4c0313a0 con 0x7f7a5c105c40 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f7a48037860 msgr2=0x7f7a48039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f7a48037860 0x7f7a48039d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a440098c0 tx=0x7f7a44006db0 comp rx=0 tx=0).stop 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 msgr2=0x7f7a5c199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c199870 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c030d30 tx=0x7f7a4c030d60 comp rx=0 tx=0).stop 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 shutdown_connections 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f7a48037860 0x7f7a48039d20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 --2- 172.21.15.79:0/3633920551 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a5c105c40 0x7f7a5c199870 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 >> 172.21.15.79:0/3633920551 conn(0x7f7a5c1011c0 msgr2=0x7f7a5c101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 shutdown_connections 2024-06-01T00:45:50.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:49.887+0000 7f7a627c4640 1 -- 172.21.15.79:0/3633920551 wait complete. 2024-06-01T00:45:50.487 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling cephadm module... 2024-06-01T00:45:51.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:51 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1862914924' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.849+0000 7f4e29011640 1 Processor -- start 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.849+0000 7f4e29011640 1 -- start start 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.849+0000 7f4e29011640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2407a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.850+0000 7f4e29011640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2407a790 con 0x7f4e2407bda0 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.850+0000 7f4e22d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2407a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.850+0000 7f4e22d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2407a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56344/0 (socket says 172.21.15.79:56344) 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.850+0000 7f4e22d9d640 1 -- 172.21.15.79:0/858145612 learned_addr learned my addr 172.21.15.79:0/858145612 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.850+0000 7f4e22d9d640 1 -- 172.21.15.79:0/858145612 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2407a8d0 con 0x7f4e2407bda0 2024-06-01T00:45:51.790 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e22d9d640 1 --2- 172.21.15.79:0/858145612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2407a250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009820 tx=0x7f4e1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14617f6bc8653b6 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e21d9b640 1 -- 172.21.15.79:0/858145612 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e1402ad40 con 0x7f4e2407bda0 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e21d9b640 1 -- 172.21.15.79:0/858145612 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4e140328c0 con 0x7f4e2407bda0 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e21d9b640 1 -- 172.21.15.79:0/858145612 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e140307f0 con 0x7f4e2407bda0 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e29011640 1 -- 172.21.15.79:0/858145612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 msgr2=0x7f4e2407a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e29011640 1 --2- 172.21.15.79:0/858145612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2407a250 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009820 tx=0x7f4e1402a270 comp rx=0 tx=0).stop 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e29011640 1 -- 172.21.15.79:0/858145612 shutdown_connections 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e29011640 1 --2- 172.21.15.79:0/858145612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2407a250 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e29011640 1 -- 172.21.15.79:0/858145612 >> 172.21.15.79:0/858145612 conn(0x7f4e24101270 msgr2=0x7f4e24103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.851+0000 7f4e29011640 1 -- 172.21.15.79:0/858145612 shutdown_connections 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.852+0000 7f4e29011640 1 -- 172.21.15.79:0/858145612 wait complete. 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.852+0000 7f4e29011640 1 Processor -- start 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.852+0000 7f4e29011640 1 -- start start 2024-06-01T00:45:51.791 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.852+0000 7f4e29011640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.852+0000 7f4e29011640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2419bf50 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e22d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e22d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2419ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56350/0 (socket says 172.21.15.79:56350) 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e22d9d640 1 -- 172.21.15.79:0/1862914924 learned_addr learned my addr 172.21.15.79:0/1862914924 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e22d9d640 1 -- 172.21.15.79:0/1862914924 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e140094d0 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e22d9d640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2419ba10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4e14030d30 tx=0x7f4e14030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e14032c40 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4e14032da0 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2419c150 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e1403c710 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.853+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2419c570 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.854+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) v1 ==== 44855+0+0 (secure 0 0 0) 0x7f4e1403c890 con 0x7f4e2407bda0 2024-06-01T00:45:51.792 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.854+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2410bf70 con 0x7f4e2407bda0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.854+0000 7f4e1bfff640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f4e10037860 0x7f4e10039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.854+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4e1406a710 con 0x7f4e2407bda0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.854+0000 7f4e2259c640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f4e10037860 0x7f4e10039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.855+0000 7f4e2259c640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f4e10037860 0x7f4e10039d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0098c0 tx=0x7f4e0c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:50.859+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4e140313a0 con 0x7f4e2407bda0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.075+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4e2419ee80 con 0x7f4e2407bda0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.154+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 5) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f4e14047980 con 0x7f4e2407bda0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.155+0000 7f4e1bfff640 1 -- 172.21.15.79:0/1862914924 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4e1406a470 con 0x7f4e2407bda0 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f4e10037860 msgr2=0x7f4e10039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f4e10037860 0x7f4e10039d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0098c0 tx=0x7f4e0c006db0 comp rx=0 tx=0).stop 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 msgr2=0x7f4e2419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2419ba10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4e14030d30 tx=0x7f4e14030d60 comp rx=0 tx=0).stop 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 shutdown_connections 2024-06-01T00:45:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f4e10037860 0x7f4e10039d20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:51.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 --2- 172.21.15.79:0/1862914924 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4e2407bda0 0x7f4e2419ba10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:51.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 >> 172.21.15.79:0/1862914924 conn(0x7f4e24101270 msgr2=0x7f4e24101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:51.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 shutdown_connections 2024-06-01T00:45:51.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:51.158+0000 7f4e29011640 1 -- 172.21.15.79:0/1862914924 wait complete. 2024-06-01T00:45:52.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:52 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1862914924' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-01T00:45:52.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:52 smithi079 ceph-mon[29356]: mgrmap e5: smithi079.emylav(active, since 5s) 2024-06-01T00:45:53.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-06-01T00:45:53.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-01T00:45:53.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-01T00:45:53.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "smithi079.emylav", 2024-06-01T00:45:53.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f911ad9a640 1 Processor -- start 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f911ad9a640 1 -- start start 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f911ad9a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f911ad9a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9114105500 con 0x7f9114104b10 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f9118b36640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f9118b36640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48282/0 (socket says 172.21.15.79:48282) 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f9118b36640 1 -- 172.21.15.79:0/991886541 learned_addr learned my addr 172.21.15.79:0/991886541 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.213+0000 7f9118b36640 1 -- 172.21.15.79:0/991886541 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114105d10 con 0x7f9114104b10 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.214+0000 7f9118b36640 1 --2- 172.21.15.79:0/991886541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114104f30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9108009820 tx=0x7f910802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150794c5755268d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.214+0000 7f91137fe640 1 -- 172.21.15.79:0/991886541 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f910802ad40 con 0x7f9114104b10 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.214+0000 7f91137fe640 1 -- 172.21.15.79:0/991886541 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f91080328c0 con 0x7f9114104b10 2024-06-01T00:45:53.107 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.214+0000 7f91137fe640 1 -- 172.21.15.79:0/991886541 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91080307f0 con 0x7f9114104b10 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 -- 172.21.15.79:0/991886541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 msgr2=0x7f9114104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 --2- 172.21.15.79:0/991886541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114104f30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9108009820 tx=0x7f910802a270 comp rx=0 tx=0).stop 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 -- 172.21.15.79:0/991886541 shutdown_connections 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 --2- 172.21.15.79:0/991886541 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114104f30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 -- 172.21.15.79:0/991886541 >> 172.21.15.79:0/991886541 conn(0x7f91141000d0 msgr2=0x7f91141024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 -- 172.21.15.79:0/991886541 shutdown_connections 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 -- 172.21.15.79:0/991886541 wait complete. 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.215+0000 7f911ad9a640 1 Processor -- start 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f911ad9a640 1 -- start start 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f911ad9a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f911ad9a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9114102960 con 0x7f9114104b10 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9118b36640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9118b36640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114102420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48288/0 (socket says 172.21.15.79:48288) 2024-06-01T00:45:53.108 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9118b36640 1 -- 172.21.15.79:0/3897140636 learned_addr learned my addr 172.21.15.79:0/3897140636 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9118b36640 1 -- 172.21.15.79:0/3897140636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91080094d0 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9118b36640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114102420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9108009940 tx=0x7f9108032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9111ffb640 1 -- 172.21.15.79:0/3897140636 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9108041070 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9111ffb640 1 -- 172.21.15.79:0/3897140636 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9108030cc0 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9114100ba0 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.216+0000 7f9111ffb640 1 -- 172.21.15.79:0/3897140636 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f910803c690 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.217+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9114100fc0 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.217+0000 7f9111ffb640 1 -- 172.21.15.79:0/3897140636 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f910803c7f0 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.218+0000 7f9111ffb640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 0x7f9104039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.218+0000 7f9111ffb640 1 -- 172.21.15.79:0/3897140636 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f910806a510 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.218+0000 7f9113fff640 1 -- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 msgr2=0x7f9104039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/616716500 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.218+0000 7f9113fff640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 0x7f9104039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.218+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9114101a80 con 0x7f9114104b10 2024-06-01T00:45:53.109 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.222+0000 7f9111ffb640 1 -- 172.21.15.79:0/3897140636 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9108031db0 con 0x7f9114104b10 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.419+0000 7f9113fff640 1 -- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 msgr2=0x7f9104039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/616716500 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.419+0000 7f9113fff640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 0x7f9104039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.448+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9114061ae0 con 0x7f9114104b10 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.448+0000 7f9111ffb640 1 -- 172.21.15.79:0/3897140636 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f91080313a0 con 0x7f9114104b10 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 msgr2=0x7f9104039d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 0x7f9104039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 msgr2=0x7f9114102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114102420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9108009940 tx=0x7f9108032c60 comp rx=0 tx=0).stop 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 shutdown_connections 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f91040378b0 0x7f9104039d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 --2- 172.21.15.79:0/3897140636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9114104b10 0x7f9114102420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 >> 172.21.15.79:0/3897140636 conn(0x7f91141000d0 msgr2=0x7f9114190650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 shutdown_connections 2024-06-01T00:45:53.110 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:52.451+0000 7f911ad9a640 1 -- 172.21.15.79:0/3897140636 wait complete. 2024-06-01T00:45:53.111 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-06-01T00:45:53.111 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 5... 2024-06-01T00:45:53.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:53 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3897140636' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-01T00:45:56.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: Active manager daemon smithi079.emylav restarted 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: Activating manager daemon smithi079.emylav 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: osdmap e2: 0 total, 0 up, 0 in 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: mgrmap e6: smithi079.emylav(active, starting, since 0.00326009s) 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr metadata", "who": "smithi079.emylav", "id": "smithi079.emylav"}]: dispatch 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: Manager daemon smithi079.emylav is now available 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:45:56.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:45:56.360 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:45:56.360 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/mirror_snapshot_schedule"}]: dispatch 2024-06-01T00:45:56.360 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:56 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/trash_purge_schedule"}]: dispatch 2024-06-01T00:45:57.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:57 smithi079 ceph-mon[29356]: Found migration_current of "None". Setting to last migration. 2024-06-01T00:45:57.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:57 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:45:57.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:57 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:45:57.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:57 smithi079 ceph-mon[29356]: mgrmap e7: smithi079.emylav(active, since 1.00569s) 2024-06-01T00:45:57.671 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-06-01T00:45:57.671 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-01T00:45:57.671 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-01T00:45:57.671 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-06-01T00:45:57.671 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.535+0000 7f122b9a5640 1 Processor -- start 2024-06-01T00:45:57.671 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.535+0000 7f122b9a5640 1 -- start start 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.536+0000 7f122b9a5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.536+0000 7f122b9a5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1224079e40 con 0x7f1224106680 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.536+0000 7f1229741640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.536+0000 7f1229741640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48300/0 (socket says 172.21.15.79:48300) 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.536+0000 7f1229741640 1 -- 172.21.15.79:0/2646607598 learned_addr learned my addr 172.21.15.79:0/2646607598 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.536+0000 7f1229741640 1 -- 172.21.15.79:0/2646607598 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1224108fb0 con 0x7f1224106680 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.537+0000 7f1229741640 1 --2- 172.21.15.79:0/2646607598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224108a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1220009820 tx=0x7f122002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20e6958d8174c8 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.537+0000 7f121bfff640 1 -- 172.21.15.79:0/2646607598 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f122002ad40 con 0x7f1224106680 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.537+0000 7f121bfff640 1 -- 172.21.15.79:0/2646607598 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f12200328c0 con 0x7f1224106680 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.537+0000 7f121bfff640 1 -- 172.21.15.79:0/2646607598 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12200307f0 con 0x7f1224106680 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.537+0000 7f122b9a5640 1 -- 172.21.15.79:0/2646607598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 msgr2=0x7f1224108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:57.672 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.537+0000 7f122b9a5640 1 --2- 172.21.15.79:0/2646607598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224108a70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1220009820 tx=0x7f122002a270 comp rx=0 tx=0).stop 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 -- 172.21.15.79:0/2646607598 shutdown_connections 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 --2- 172.21.15.79:0/2646607598 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224108a70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 -- 172.21.15.79:0/2646607598 >> 172.21.15.79:0/2646607598 conn(0x7f12241000d0 msgr2=0x7f12241024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 -- 172.21.15.79:0/2646607598 shutdown_connections 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 -- 172.21.15.79:0/2646607598 wait complete. 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 Processor -- start 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 -- start start 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f122b9a5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1224199d80 con 0x7f1224106680 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f1229741640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f1229741640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48302/0 (socket says 172.21.15.79:48302) 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.538+0000 7f1229741640 1 -- 172.21.15.79:0/3803507626 learned_addr learned my addr 172.21.15.79:0/3803507626 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:57.673 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.539+0000 7f1229741640 1 -- 172.21.15.79:0/3803507626 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12200094d0 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.539+0000 7f1229741640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224199840 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1220030d30 tx=0x7f1220030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.539+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1220032c40 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.539+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1220032da0 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.539+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f122003c710 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.539+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1224199f80 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.539+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122419a3a0 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.541+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f122003d480 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.541+0000 7f121a7fc640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 0x7f1210039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.541+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 --> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f121003a480 con 0x7f12100378b0 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.541+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f122006a7c0 con 0x7f1224106680 2024-06-01T00:45:57.674 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.541+0000 7f1228f40640 1 -- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 msgr2=0x7f1210039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/616716500 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.541+0000 7f1228f40640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 0x7f1210039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.742+0000 7f1228f40640 1 -- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 msgr2=0x7f1210039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/616716500 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:53.742+0000 7f1228f40640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 0x7f1210039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:54.143+0000 7f1228f40640 1 -- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 msgr2=0x7f1210039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/616716500 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:54.143+0000 7f1228f40640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 0x7f1210039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:54.944+0000 7f1228f40640 1 -- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 msgr2=0x7f1210039d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/616716500 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:54.944+0000 7f1228f40640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 0x7f1210039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:56.059+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 6) v1 ==== 44633+0+0 (secure 0 0 0) 0x7f122003cdb0 con 0x7f1224106680 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:56.059+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 msgr2=0x7f1210039d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:56.059+0000 7f121a7fc640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 0x7f1210039d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.061+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 7) v1 ==== 44760+0+0 (secure 0 0 0) 0x7f122003bb20 con 0x7f1224106680 2024-06-01T00:45:57.675 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.061+0000 7f121a7fc640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f121003afc0 0x7f121003d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.061+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f121003a480 con 0x7f121003afc0 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.062+0000 7f1228f40640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f121003afc0 0x7f121003d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.062+0000 7f1228f40640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f121003afc0 0x7f121003d3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1214003980 tx=0x7f12140092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.063+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f121003a480 con 0x7f121003afc0 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.068+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1224061ae0 con 0x7f121003afc0 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f121a7fc640 1 -- 172.21.15.79:0/3803507626 <== mgr.14118 v2:172.21.15.79:6800/1904937508 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f1224061ae0 con 0x7f121003afc0 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f121003afc0 msgr2=0x7f121003d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f121003afc0 0x7f121003d3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1214003980 tx=0x7f12140092b0 comp rx=0 tx=0).stop 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 msgr2=0x7f1224199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224199840 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1220030d30 tx=0x7f1220030d60 comp rx=0 tx=0).stop 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 shutdown_connections 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f121003afc0 0x7f121003d3b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:6800/616716500,v1:172.21.15.79:6801/616716500] conn(0x7f12100378b0 0x7f1210039d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:57.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 --2- 172.21.15.79:0/3803507626 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1224106680 0x7f1224199840 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:57.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 >> 172.21.15.79:0/3803507626 conn(0x7f12241000d0 msgr2=0x7f1224100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:57.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 shutdown_connections 2024-06-01T00:45:57.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:57.069+0000 7f122b9a5640 1 -- 172.21.15.79:0/3803507626 wait complete. 2024-06-01T00:45:57.677 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 5 is available 2024-06-01T00:45:57.677 INFO:teuthology.orchestra.run.smithi079.stdout:Setting orchestrator backend to cephadm... 2024-06-01T00:45:58.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:58 smithi079 ceph-mon[29356]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-01T00:45:58.611 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:58 smithi079 ceph-mon[29356]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-01T00:45:58.611 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:58 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:45:58.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2db0dae640 1 Processor -- start 2024-06-01T00:45:58.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2db0dae640 1 -- start start 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2db0dae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac108b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2db0dae640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac079d80 con 0x7f2dac106740 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2daa59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac108b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2daa59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac108b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48394/0 (socket says 172.21.15.79:48394) 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2daa59c640 1 -- 172.21.15.79:0/2204785488 learned_addr learned my addr 172.21.15.79:0/2204785488 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.110+0000 7f2daa59c640 1 -- 172.21.15.79:0/2204785488 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dac079ec0 con 0x7f2dac106740 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.111+0000 7f2daa59c640 1 --2- 172.21.15.79:0/2204785488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac108b30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009820 tx=0x7f2d9c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d98a08c284ea14 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.112+0000 7f2da9d9b640 1 -- 172.21.15.79:0/2204785488 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9c02ad40 con 0x7f2dac106740 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.112+0000 7f2da9d9b640 1 -- 172.21.15.79:0/2204785488 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2d9c0328c0 con 0x7f2dac106740 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.112+0000 7f2da9d9b640 1 -- 172.21.15.79:0/2204785488 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9c0307f0 con 0x7f2dac106740 2024-06-01T00:45:58.877 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.113+0000 7f2db0dae640 1 -- 172.21.15.79:0/2204785488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 msgr2=0x7f2dac108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.113+0000 7f2db0dae640 1 --2- 172.21.15.79:0/2204785488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac108b30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009820 tx=0x7f2d9c02a270 comp rx=0 tx=0).stop 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.113+0000 7f2db0dae640 1 -- 172.21.15.79:0/2204785488 shutdown_connections 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.113+0000 7f2db0dae640 1 --2- 172.21.15.79:0/2204785488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac108b30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.113+0000 7f2db0dae640 1 -- 172.21.15.79:0/2204785488 >> 172.21.15.79:0/2204785488 conn(0x7f2dac1000d0 msgr2=0x7f2dac1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.113+0000 7f2db0dae640 1 -- 172.21.15.79:0/2204785488 shutdown_connections 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.114+0000 7f2db0dae640 1 -- 172.21.15.79:0/2204785488 wait complete. 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.114+0000 7f2db0dae640 1 Processor -- start 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.114+0000 7f2db0dae640 1 -- start start 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.114+0000 7f2db0dae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:58.878 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.114+0000 7f2db0dae640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dac199d70 con 0x7f2dac106740 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2daa59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2daa59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48398/0 (socket says 172.21.15.79:48398) 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2daa59c640 1 -- 172.21.15.79:0/1545146747 learned_addr learned my addr 172.21.15.79:0/1545146747 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2daa59c640 1 -- 172.21.15.79:0/1545146747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9c0094d0 con 0x7f2dac106740 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2daa59c640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac199830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c030d30 tx=0x7f2d9c030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2da37fe640 1 -- 172.21.15.79:0/1545146747 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9c032c40 con 0x7f2dac106740 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2da37fe640 1 -- 172.21.15.79:0/1545146747 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2d9c032da0 con 0x7f2dac106740 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dac199f70 con 0x7f2dac106740 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2da37fe640 1 -- 172.21.15.79:0/1545146747 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d9c03c710 con 0x7f2dac106740 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.115+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dac19a390 con 0x7f2dac106740 2024-06-01T00:45:58.879 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.116+0000 7f2da37fe640 1 -- 172.21.15.79:0/1545146747 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f2d9c03c980 con 0x7f2dac106740 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.117+0000 7f2da37fe640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2d98037860 0x7f2d98039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.117+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dac109d20 con 0x7f2dac106740 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.117+0000 7f2da37fe640 1 -- 172.21.15.79:0/1545146747 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2d9c06a7b0 con 0x7f2dac106740 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.117+0000 7f2da3fff640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2d98037860 0x7f2d98039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.117+0000 7f2da3fff640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2d98037860 0x7f2d98039d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d940098c0 tx=0x7f2d94006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.121+0000 7f2da37fe640 1 -- 172.21.15.79:0/1545146747 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2d9c0313a0 con 0x7f2dac106740 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.307+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dac1011f0 con 0x7f2d98037860 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.316+0000 7f2da37fe640 1 -- 172.21.15.79:0/1545146747 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2dac1011f0 con 0x7f2d98037860 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2d98037860 msgr2=0x7f2d98039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2d98037860 0x7f2d98039d20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d940098c0 tx=0x7f2d94006db0 comp rx=0 tx=0).stop 2024-06-01T00:45:58.880 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 msgr2=0x7f2dac199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:45:58.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac199830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c030d30 tx=0x7f2d9c030d60 comp rx=0 tx=0).stop 2024-06-01T00:45:58.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 shutdown_connections 2024-06-01T00:45:58.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2d98037860 0x7f2d98039d20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:58.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 --2- 172.21.15.79:0/1545146747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2dac106740 0x7f2dac199830 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:45:58.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 >> 172.21.15.79:0/1545146747 conn(0x7f2dac1000d0 msgr2=0x7f2dac100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:45:58.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 shutdown_connections 2024-06-01T00:45:58.881 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:58.318+0000 7f2db0dae640 1 -- 172.21.15.79:0/1545146747 wait complete. 2024-06-01T00:45:59.286 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: [01/Jun/2024:00:45:57] ENGINE Bus STARTING 2024-06-01T00:45:59.286 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: [01/Jun/2024:00:45:57] ENGINE Serving on http://172.21.15.79:8765 2024-06-01T00:45:59.286 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: [01/Jun/2024:00:45:57] ENGINE Serving on https://172.21.15.79:7150 2024-06-01T00:45:59.286 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: [01/Jun/2024:00:45:57] ENGINE Bus STARTED 2024-06-01T00:45:59.286 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: [01/Jun/2024:00:45:57] ENGINE Error in HTTPServer.serve 2024-06-01T00:45:59.286 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Traceback (most recent call last): 2024-06-01T00:45:59.286 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._run(expiration_interval) 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: s = self.context.wrap_socket( 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-01T00:45:59.287 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: return self.sslsocket_class._create( 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self.do_handshake() 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._sslobj.do_handshake() 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: mgrmap e8: smithi079.emylav(active, since 2s) 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:45:59.288 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:45:59 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:46:00.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-01T00:46:00.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.284+0000 7fd8b47e2640 1 Processor -- start 2024-06-01T00:46:00.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.284+0000 7fd8b47e2640 1 -- start start 2024-06-01T00:46:00.096 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.285+0000 7fd8b47e2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:00.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.285+0000 7fd8b47e2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac105500 con 0x7fd8ac104b10 2024-06-01T00:46:00.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.285+0000 7fd8b257e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:00.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.285+0000 7fd8b257e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48402/0 (socket says 172.21.15.79:48402) 2024-06-01T00:46:00.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.285+0000 7fd8b257e640 1 -- 172.21.15.79:0/2043571413 learned_addr learned my addr 172.21.15.79:0/2043571413 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:00.097 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.285+0000 7fd8b257e640 1 -- 172.21.15.79:0/2043571413 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac105d10 con 0x7fd8ac104b10 2024-06-01T00:46:00.098 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.286+0000 7fd8b257e640 1 --2- 172.21.15.79:0/2043571413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac104f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009820 tx=0x7fd8a802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24f80bd99dee2f5 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:00.098 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.286+0000 7fd8b157c640 1 -- 172.21.15.79:0/2043571413 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8a802ad40 con 0x7fd8ac104b10 2024-06-01T00:46:00.098 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.286+0000 7fd8b157c640 1 -- 172.21.15.79:0/2043571413 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd8a80328c0 con 0x7fd8ac104b10 2024-06-01T00:46:00.098 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.286+0000 7fd8b157c640 1 -- 172.21.15.79:0/2043571413 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8a80307f0 con 0x7fd8ac104b10 2024-06-01T00:46:00.098 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.287+0000 7fd8b47e2640 1 -- 172.21.15.79:0/2043571413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 msgr2=0x7fd8ac104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:00.099 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.287+0000 7fd8b47e2640 1 --2- 172.21.15.79:0/2043571413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac104f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009820 tx=0x7fd8a802a270 comp rx=0 tx=0).stop 2024-06-01T00:46:00.099 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.287+0000 7fd8b47e2640 1 -- 172.21.15.79:0/2043571413 shutdown_connections 2024-06-01T00:46:00.099 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.287+0000 7fd8b47e2640 1 --2- 172.21.15.79:0/2043571413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac104f30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:00.099 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.287+0000 7fd8b47e2640 1 -- 172.21.15.79:0/2043571413 >> 172.21.15.79:0/2043571413 conn(0x7fd8ac1000d0 msgr2=0x7fd8ac1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:00.099 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.287+0000 7fd8b47e2640 1 -- 172.21.15.79:0/2043571413 shutdown_connections 2024-06-01T00:46:00.099 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.287+0000 7fd8b47e2640 1 -- 172.21.15.79:0/2043571413 wait complete. 2024-06-01T00:46:00.100 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b47e2640 1 Processor -- start 2024-06-01T00:46:00.100 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b47e2640 1 -- start start 2024-06-01T00:46:00.100 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b47e2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:00.100 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b47e2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac198aa0 con 0x7fd8ac104b10 2024-06-01T00:46:00.100 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b257e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:00.101 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b257e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48416/0 (socket says 172.21.15.79:48416) 2024-06-01T00:46:00.101 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b257e640 1 -- 172.21.15.79:0/4213126049 learned_addr learned my addr 172.21.15.79:0/4213126049 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:00.101 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b257e640 1 -- 172.21.15.79:0/4213126049 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a80094d0 con 0x7fd8ac104b10 2024-06-01T00:46:00.101 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.288+0000 7fd8b257e640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac198560 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd8a802a790 tx=0x7fd8a8030ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:00.101 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.289+0000 7fd8a37fe640 1 -- 172.21.15.79:0/4213126049 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8a8032740 con 0x7fd8ac104b10 2024-06-01T00:46:00.101 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.289+0000 7fd8a37fe640 1 -- 172.21.15.79:0/4213126049 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd8a802ae00 con 0x7fd8ac104b10 2024-06-01T00:46:00.102 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.289+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac198ca0 con 0x7fd8ac104b10 2024-06-01T00:46:00.102 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.289+0000 7fd8a37fe640 1 -- 172.21.15.79:0/4213126049 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8a803d8e0 con 0x7fd8ac104b10 2024-06-01T00:46:00.102 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.289+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac1990c0 con 0x7fd8ac104b10 2024-06-01T00:46:00.102 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.289+0000 7fd8a37fe640 1 -- 172.21.15.79:0/4213126049 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fd8a803da40 con 0x7fd8ac104b10 2024-06-01T00:46:00.102 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.290+0000 7fd8a37fe640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fd8980378b0 0x7fd898039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:00.103 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.290+0000 7fd8a37fe640 1 -- 172.21.15.79:0/4213126049 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd8a806aa90 con 0x7fd8ac104b10 2024-06-01T00:46:00.103 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.290+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ac061ae0 con 0x7fd8ac104b10 2024-06-01T00:46:00.103 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.290+0000 7fd8b1d7d640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fd8980378b0 0x7fd898039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:00.103 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.290+0000 7fd8b1d7d640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fd8980378b0 0x7fd898039d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd89c0098c0 tx=0x7fd89c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:00.103 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.295+0000 7fd8a37fe640 1 -- 172.21.15.79:0/4213126049 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd8a80313a0 con 0x7fd8ac104b10 2024-06-01T00:46:00.104 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.469+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8ac101020 con 0x7fd8980378b0 2024-06-01T00:46:00.104 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.470+0000 7fd8a37fe640 1 -- 172.21.15.79:0/4213126049 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd8ac101020 con 0x7fd8980378b0 2024-06-01T00:46:00.104 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.471+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fd8980378b0 msgr2=0x7fd898039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:00.104 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.471+0000 7fd8b47e2640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fd8980378b0 0x7fd898039d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd89c0098c0 tx=0x7fd89c006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:00.104 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 msgr2=0x7fd8ac198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:00.104 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac198560 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd8a802a790 tx=0x7fd8a8030ce0 comp rx=0 tx=0).stop 2024-06-01T00:46:00.105 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 shutdown_connections 2024-06-01T00:46:00.105 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fd8980378b0 0x7fd898039d70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:00.105 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 --2- 172.21.15.79:0/4213126049 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8ac104b10 0x7fd8ac198560 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:00.105 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 >> 172.21.15.79:0/4213126049 conn(0x7fd8ac1000d0 msgr2=0x7fd8ac100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:00.105 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 shutdown_connections 2024-06-01T00:46:00.106 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:45:59.472+0000 7fd8b47e2640 1 -- 172.21.15.79:0/4213126049 wait complete. 2024-06-01T00:46:00.106 INFO:teuthology.orchestra.run.smithi079.stdout:Generating ssh key... 2024-06-01T00:46:00.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:00 smithi079 ceph-mon[29356]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:01.239 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:01 smithi079 ceph-mon[29356]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:01.239 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:01 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:01.239 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:01 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.480+0000 7faaa08c2640 1 Processor -- start 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.480+0000 7faaa08c2640 1 -- start start 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.480+0000 7faaa08c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.480+0000 7faaa08c2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c105500 con 0x7faa9c104b10 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.481+0000 7faa9a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.481+0000 7faa9a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48418/0 (socket says 172.21.15.79:48418) 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.481+0000 7faa9a59c640 1 -- 172.21.15.79:0/1510588376 learned_addr learned my addr 172.21.15.79:0/1510588376 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:01.514 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.481+0000 7faa9a59c640 1 -- 172.21.15.79:0/1510588376 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c105d10 con 0x7faa9c104b10 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faa9a59c640 1 --2- 172.21.15.79:0/1510588376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c104f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009820 tx=0x7faa8c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db9c07d2d31eadbf server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faa99d9b640 1 -- 172.21.15.79:0/1510588376 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c02ad40 con 0x7faa9c104b10 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faa99d9b640 1 -- 172.21.15.79:0/1510588376 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7faa8c0328c0 con 0x7faa9c104b10 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faa99d9b640 1 -- 172.21.15.79:0/1510588376 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c0307f0 con 0x7faa9c104b10 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faaa08c2640 1 -- 172.21.15.79:0/1510588376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 msgr2=0x7faa9c104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faaa08c2640 1 --2- 172.21.15.79:0/1510588376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c104f30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009820 tx=0x7faa8c02a270 comp rx=0 tx=0).stop 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faaa08c2640 1 -- 172.21.15.79:0/1510588376 shutdown_connections 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faaa08c2640 1 --2- 172.21.15.79:0/1510588376 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c104f30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faaa08c2640 1 -- 172.21.15.79:0/1510588376 >> 172.21.15.79:0/1510588376 conn(0x7faa9c1000d0 msgr2=0x7faa9c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faaa08c2640 1 -- 172.21.15.79:0/1510588376 shutdown_connections 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.482+0000 7faaa08c2640 1 -- 172.21.15.79:0/1510588376 wait complete. 2024-06-01T00:46:01.515 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faaa08c2640 1 Processor -- start 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faaa08c2640 1 -- start start 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faaa08c2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c0799f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faaa08c2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c079f30 con 0x7faa9c104b10 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faa9a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c0799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faa9a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c0799f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48426/0 (socket says 172.21.15.79:48426) 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faa9a59c640 1 -- 172.21.15.79:0/4285300997 learned_addr learned my addr 172.21.15.79:0/4285300997 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.483+0000 7faa9a59c640 1 -- 172.21.15.79:0/4285300997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0094d0 con 0x7faa9c104b10 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.484+0000 7faa9a59c640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c0799f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa8c030d30 tx=0x7faa8c030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.484+0000 7faa937fe640 1 -- 172.21.15.79:0/4285300997 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c032c40 con 0x7faa9c104b10 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.484+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c19c550 con 0x7faa9c104b10 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.484+0000 7faa937fe640 1 -- 172.21.15.79:0/4285300997 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7faa8c032da0 con 0x7faa9c104b10 2024-06-01T00:46:01.516 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.484+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c19c970 con 0x7faa9c104b10 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.484+0000 7faa937fe640 1 -- 172.21.15.79:0/4285300997 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa8c03c6a0 con 0x7faa9c104b10 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.485+0000 7faa937fe640 1 -- 172.21.15.79:0/4285300997 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7faa8c03d480 con 0x7faa9c104b10 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.485+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c061ae0 con 0x7faa9c104b10 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.485+0000 7faa937fe640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7faa88037860 0x7faa88039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.485+0000 7faa937fe640 1 -- 172.21.15.79:0/4285300997 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faa8c06a9b0 con 0x7faa9c104b10 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.485+0000 7faa93fff640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7faa88037860 0x7faa88039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.485+0000 7faa93fff640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7faa88037860 0x7faa88039d20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa840098c0 tx=0x7faa84006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.490+0000 7faa937fe640 1 -- 172.21.15.79:0/4285300997 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7faa8c03d730 con 0x7faa9c104b10 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.661+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7faa9c108f10 con 0x7faa88037860 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.938+0000 7faa937fe640 1 -- 172.21.15.79:0/4285300997 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faa9c108f10 con 0x7faa88037860 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7faa88037860 msgr2=0x7faa88039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7faa88037860 0x7faa88039d20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa840098c0 tx=0x7faa84006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 msgr2=0x7faa9c0799f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:01.517 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c0799f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa8c030d30 tx=0x7faa8c030d60 comp rx=0 tx=0).stop 2024-06-01T00:46:01.518 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 shutdown_connections 2024-06-01T00:46:01.518 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7faa88037860 0x7faa88039d20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:01.518 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 --2- 172.21.15.79:0/4285300997 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faa9c104b10 0x7faa9c0799f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:01.518 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 >> 172.21.15.79:0/4285300997 conn(0x7faa9c1000d0 msgr2=0x7faa9c100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:01.518 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 shutdown_connections 2024-06-01T00:46:01.518 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:00.940+0000 7faaa08c2640 1 -- 172.21.15.79:0/4285300997 wait complete. 2024-06-01T00:46:02.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:02 smithi079 ceph-mon[29356]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:02.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:02 smithi079 ceph-mon[29356]: Generating ssh key... 2024-06-01T00:46:02.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR44OWYFqe2zC5QDBHiEBFZwZ0PrBG3RCk1GPVU/sn/Sbbq4wnAt+LODM/Wr94Wxs9C/5dLJ3LPa+MiYySxqqmTpEzEuT8KxghK/37azfQi3ETtCFRrOxks0BS/070PXUa7eRcxbXucE1qJJXB2xbfmGbjZC43pI0D4g7n0bb7K1/SQNzOeAWTAhJLDrxUFi6wJWj+0Xbi8h10EaatoXC9I9KX9iKv69hAHlbv8uEkd9Tv7MjZzQhAK5+VTG97lAyjUv4+LO5jp8MRAECgV/RXr1aU8YNWUoi6Ohrv1Zeo7DOO9QuFF2MouoCj+6QV9IDHytnQ0t0h4+EKng9HJb6MuSm00u/t49UekfDCXF/bLa0IqMzRRslOGbtgcXy2rm9wXlLqloLR3YjHirxRBQ8G6Vm9xmHS1aTK5HBH+b64IVHBYpE2DVIaS5v2BM3D4+CUMKgtBoX8fw59wNEPqZukm16Jv3FuIsFsPrK8FCD/eaeuwNrP3Bwo2yeh6fk6uYM= ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:46:02.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.886+0000 7fde25d27640 1 Processor -- start 2024-06-01T00:46:02.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.886+0000 7fde25d27640 1 -- start start 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.887+0000 7fde25d27640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde201048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.887+0000 7fde25d27640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde20079de0 con 0x7fde201044d0 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.887+0000 7fde1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde201048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.887+0000 7fde1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde201048f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48440/0 (socket says 172.21.15.79:48440) 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.887+0000 7fde1f7fe640 1 -- 172.21.15.79:0/320697130 learned_addr learned my addr 172.21.15.79:0/320697130 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.887+0000 7fde1f7fe640 1 -- 172.21.15.79:0/320697130 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde20104e30 con 0x7fde201044d0 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.888+0000 7fde1f7fe640 1 --2- 172.21.15.79:0/320697130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde201048f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde10009820 tx=0x7fde1002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0517beca9fa99e9 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.888+0000 7fde1e7fc640 1 -- 172.21.15.79:0/320697130 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde1002ad40 con 0x7fde201044d0 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.888+0000 7fde1e7fc640 1 -- 172.21.15.79:0/320697130 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fde100328c0 con 0x7fde201044d0 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.888+0000 7fde1e7fc640 1 -- 172.21.15.79:0/320697130 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde100307f0 con 0x7fde201044d0 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.888+0000 7fde25d27640 1 -- 172.21.15.79:0/320697130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 msgr2=0x7fde201048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.888+0000 7fde25d27640 1 --2- 172.21.15.79:0/320697130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde201048f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde10009820 tx=0x7fde1002a270 comp rx=0 tx=0).stop 2024-06-01T00:46:02.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 -- 172.21.15.79:0/320697130 shutdown_connections 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 --2- 172.21.15.79:0/320697130 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde201048f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 -- 172.21.15.79:0/320697130 >> 172.21.15.79:0/320697130 conn(0x7fde201000d0 msgr2=0x7fde201024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 -- 172.21.15.79:0/320697130 shutdown_connections 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 -- 172.21.15.79:0/320697130 wait complete. 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 Processor -- start 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 -- start start 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.889+0000 7fde25d27640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde2019ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde25d27640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2019bf70 con 0x7fde201044d0 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde2019ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde2019ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48442/0 (socket says 172.21.15.79:48442) 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1f7fe640 1 -- 172.21.15.79:0/2579570594 learned_addr learned my addr 172.21.15.79:0/2579570594 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:02.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1f7fe640 1 -- 172.21.15.79:0/2579570594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100094d0 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1f7fe640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde2019ba30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde10030d30 tx=0x7fde10030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1cff9640 1 -- 172.21.15.79:0/2579570594 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde10032c40 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1cff9640 1 -- 172.21.15.79:0/2579570594 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fde10032da0 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2019c170 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde1cff9640 1 -- 172.21.15.79:0/2579570594 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde1003c710 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.890+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2019c590 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.891+0000 7fde1cff9640 1 -- 172.21.15.79:0/2579570594 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fde1003c980 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.891+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde20195720 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.891+0000 7fde1cff9640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fde08037860 0x7fde08039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.891+0000 7fde1cff9640 1 -- 172.21.15.79:0/2579570594 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fde1006a7a0 con 0x7fde201044d0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.891+0000 7fde1effd640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fde08037860 0x7fde08039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.892+0000 7fde1effd640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fde08037860 0x7fde08039d20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fde140098c0 tx=0x7fde14006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:02.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:01.896+0000 7fde1cff9640 1 -- 172.21.15.79:0/2579570594 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fde100313a0 con 0x7fde201044d0 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.068+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fde2002c5a0 con 0x7fde08037860 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.069+0000 7fde1cff9640 1 -- 172.21.15.79:0/2579570594 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fde2002c5a0 con 0x7fde08037860 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fde08037860 msgr2=0x7fde08039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fde08037860 0x7fde08039d20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fde140098c0 tx=0x7fde14006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 msgr2=0x7fde2019ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde2019ba30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde10030d30 tx=0x7fde10030d60 comp rx=0 tx=0).stop 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 shutdown_connections 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fde08037860 0x7fde08039d20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 --2- 172.21.15.79:0/2579570594 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde201044d0 0x7fde2019ba30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 >> 172.21.15.79:0/2579570594 conn(0x7fde201000d0 msgr2=0x7fde201928f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 shutdown_connections 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:02.071+0000 7fde25d27640 1 -- 172.21.15.79:0/2579570594 wait complete. 2024-06-01T00:46:02.640 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-01T00:46:02.641 INFO:teuthology.orchestra.run.smithi079.stdout:Adding key to root@localhost authorized_keys... 2024-06-01T00:46:02.641 INFO:teuthology.orchestra.run.smithi079.stdout:Adding host smithi079... 2024-06-01T00:46:03.214 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:03 smithi079 ceph-mon[29356]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:04.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:04 smithi079 ceph-mon[29356]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:04.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:04 smithi079 ceph-mon[29356]: Deploying cephadm binary to smithi079 2024-06-01T00:46:06.522 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Added host 'smithi079' with addr '172.21.15.79' 2024-06-01T00:46:06.522 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.026+0000 7f1997d05640 1 Processor -- start 2024-06-01T00:46:06.522 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.026+0000 7f1997d05640 1 -- start start 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.026+0000 7f1997d05640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f19901060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.026+0000 7f1997d05640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1990106680 con 0x7f1990105c90 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.027+0000 7f1995aa1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f19901060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.027+0000 7f1995aa1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f19901060b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56870/0 (socket says 172.21.15.79:56870) 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.027+0000 7f1995aa1640 1 -- 172.21.15.79:0/1112605096 learned_addr learned my addr 172.21.15.79:0/1112605096 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.027+0000 7f1995aa1640 1 -- 172.21.15.79:0/1112605096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990106e40 con 0x7f1990105c90 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1995aa1640 1 --2- 172.21.15.79:0/1112605096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f19901060b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f198c009820 tx=0x7f198c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fd46323e249c02 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1994a9f640 1 -- 172.21.15.79:0/1112605096 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c02ad40 con 0x7f1990105c90 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1994a9f640 1 -- 172.21.15.79:0/1112605096 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f198c0328c0 con 0x7f1990105c90 2024-06-01T00:46:06.523 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1994a9f640 1 -- 172.21.15.79:0/1112605096 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c0307f0 con 0x7f1990105c90 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1997d05640 1 -- 172.21.15.79:0/1112605096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 msgr2=0x7f19901060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1997d05640 1 --2- 172.21.15.79:0/1112605096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f19901060b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f198c009820 tx=0x7f198c02a270 comp rx=0 tx=0).stop 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1997d05640 1 -- 172.21.15.79:0/1112605096 shutdown_connections 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1997d05640 1 --2- 172.21.15.79:0/1112605096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f19901060b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1997d05640 1 -- 172.21.15.79:0/1112605096 >> 172.21.15.79:0/1112605096 conn(0x7f1990101270 msgr2=0x7f1990103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1997d05640 1 -- 172.21.15.79:0/1112605096 shutdown_connections 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.028+0000 7f1997d05640 1 -- 172.21.15.79:0/1112605096 wait complete. 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1997d05640 1 Processor -- start 2024-06-01T00:46:06.524 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1997d05640 1 -- start start 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1997d05640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f1990199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1997d05640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1990199db0 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1995aa1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f1990199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1995aa1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f1990199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56872/0 (socket says 172.21.15.79:56872) 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1995aa1640 1 -- 172.21.15.79:0/3430069698 learned_addr learned my addr 172.21.15.79:0/3430069698 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.029+0000 7f1995aa1640 1 -- 172.21.15.79:0/3430069698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0094d0 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.030+0000 7f1995aa1640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f1990199870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f198c030d30 tx=0x7f198c030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.030+0000 7f1986ffd640 1 -- 172.21.15.79:0/3430069698 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c032c40 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.030+0000 7f1986ffd640 1 -- 172.21.15.79:0/3430069698 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f198c032da0 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.030+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1990199fb0 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.030+0000 7f1986ffd640 1 -- 172.21.15.79:0/3430069698 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198c03c6a0 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.030+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199019a3d0 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.031+0000 7f1986ffd640 1 -- 172.21.15.79:0/3430069698 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f198c03d480 con 0x7f1990105c90 2024-06-01T00:46:06.525 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.031+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1990061ae0 con 0x7f1990105c90 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.031+0000 7f1986ffd640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f197c0378b0 0x7f197c039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.031+0000 7f1986ffd640 1 -- 172.21.15.79:0/3430069698 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f198c06a970 con 0x7f1990105c90 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.031+0000 7f19952a0640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f197c0378b0 0x7f197c039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.031+0000 7f19952a0640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f197c0378b0 0x7f197c039d70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19800098c0 tx=0x7f1980006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.036+0000 7f1986ffd640 1 -- 172.21.15.79:0/3430069698 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f198c031580 con 0x7f1990105c90 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:03.212+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}) v1 -- 0x7f19901023d0 con 0x7f197c0378b0 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.846+0000 7f1986ffd640 1 -- 172.21.15.79:0/3430069698 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f19901023d0 con 0x7f197c0378b0 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f197c0378b0 msgr2=0x7f197c039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f197c0378b0 0x7f197c039d70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19800098c0 tx=0x7f1980006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 msgr2=0x7f1990199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f1990199870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f198c030d30 tx=0x7f198c030d60 comp rx=0 tx=0).stop 2024-06-01T00:46:06.526 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 shutdown_connections 2024-06-01T00:46:06.527 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f197c0378b0 0x7f197c039d70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:06.527 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 --2- 172.21.15.79:0/3430069698 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1990105c90 0x7f1990199870 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:06.527 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 >> 172.21.15.79:0/3430069698 conn(0x7f1990101270 msgr2=0x7f1990101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:06.527 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.849+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 shutdown_connections 2024-06-01T00:46:06.527 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:05.850+0000 7f1997d05640 1 -- 172.21.15.79:0/3430069698 wait complete. 2024-06-01T00:46:06.527 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying mon service with default placement... 2024-06-01T00:46:07.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:06 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:07.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:06 smithi079 ceph-mon[29356]: Added host smithi079 2024-06-01T00:46:07.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:06 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:46:08.350 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:08 smithi079 ceph-mon[29356]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:08.351 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:08 smithi079 ceph-mon[29356]: Saving service mon spec with placement count:5 2024-06-01T00:46:08.351 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:08 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:08.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-01T00:46:08.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.910+0000 7f6c5e833640 1 Processor -- start 2024-06-01T00:46:08.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.910+0000 7f6c5e833640 1 -- start start 2024-06-01T00:46:08.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.910+0000 7f6c5e833640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5807a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.910+0000 7f6c5e833640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5807a840 con 0x7f6c5807be10 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.911+0000 7f6c57fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5807a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.911+0000 7f6c57fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5807a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56878/0 (socket says 172.21.15.79:56878) 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.911+0000 7f6c57fff640 1 -- 172.21.15.79:0/2022210566 learned_addr learned my addr 172.21.15.79:0/2022210566 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.911+0000 7f6c57fff640 1 -- 172.21.15.79:0/2022210566 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5807a980 con 0x7f6c5807be10 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c57fff640 1 --2- 172.21.15.79:0/2022210566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5807a270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009820 tx=0x7f6c4802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9c6ebd91e7122d server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c56ffd640 1 -- 172.21.15.79:0/2022210566 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c4802ad40 con 0x7f6c5807be10 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c56ffd640 1 -- 172.21.15.79:0/2022210566 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6c480328c0 con 0x7f6c5807be10 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c56ffd640 1 -- 172.21.15.79:0/2022210566 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c480307f0 con 0x7f6c5807be10 2024-06-01T00:46:08.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c5e833640 1 -- 172.21.15.79:0/2022210566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 msgr2=0x7f6c5807a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:08.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c5e833640 1 --2- 172.21.15.79:0/2022210566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5807a270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009820 tx=0x7f6c4802a270 comp rx=0 tx=0).stop 2024-06-01T00:46:08.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c5e833640 1 -- 172.21.15.79:0/2022210566 shutdown_connections 2024-06-01T00:46:08.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c5e833640 1 --2- 172.21.15.79:0/2022210566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5807a270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c5e833640 1 -- 172.21.15.79:0/2022210566 >> 172.21.15.79:0/2022210566 conn(0x7f6c581011c0 msgr2=0x7f6c58103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c5e833640 1 -- 172.21.15.79:0/2022210566 shutdown_connections 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.912+0000 7f6c5e833640 1 -- 172.21.15.79:0/2022210566 wait complete. 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.913+0000 7f6c5e833640 1 Processor -- start 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.913+0000 7f6c5e833640 1 -- start start 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.913+0000 7f6c5e833640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.913+0000 7f6c5e833640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5819bf30 con 0x7f6c5807be10 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.913+0000 7f6c57fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.913+0000 7f6c57fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5819b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56884/0 (socket says 172.21.15.79:56884) 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.913+0000 7f6c57fff640 1 -- 172.21.15.79:0/1578397415 learned_addr learned my addr 172.21.15.79:0/1578397415 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.914+0000 7f6c57fff640 1 -- 172.21.15.79:0/1578397415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c480094d0 con 0x7f6c5807be10 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.914+0000 7f6c57fff640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5819b9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009940 tx=0x7f6c48032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:08.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.914+0000 7f6c557fa640 1 -- 172.21.15.79:0/1578397415 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c48032df0 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.914+0000 7f6c557fa640 1 -- 172.21.15.79:0/1578397415 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6c48030cc0 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.914+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5819c130 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.914+0000 7f6c557fa640 1 -- 172.21.15.79:0/1578397415 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c4803c6b0 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.914+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5819c550 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.915+0000 7f6c557fa640 1 -- 172.21.15.79:0/1578397415 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f6c4803c970 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.915+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5810bfc0 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.915+0000 7f6c557fa640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6c40037860 0x7f6c40039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.915+0000 7f6c557fa640 1 -- 172.21.15.79:0/1578397415 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6c4806a9d0 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.915+0000 7f6c577fe640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6c40037860 0x7f6c40039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.915+0000 7f6c577fe640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6c40037860 0x7f6c40039d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0098c0 tx=0x7f6c4c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:06.920+0000 7f6c557fa640 1 -- 172.21.15.79:0/1578397415 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6c480313a0 con 0x7f6c5807be10 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.113+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c581022b0 con 0x7f6c40037860 2024-06-01T00:46:08.638 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.117+0000 7f6c557fa640 1 -- 172.21.15.79:0/1578397415 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6c581022b0 con 0x7f6c40037860 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6c40037860 msgr2=0x7f6c40039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6c40037860 0x7f6c40039d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0098c0 tx=0x7f6c4c006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 msgr2=0x7f6c5819b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5819b9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009940 tx=0x7f6c48032c60 comp rx=0 tx=0).stop 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 shutdown_connections 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6c40037860 0x7f6c40039d20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 --2- 172.21.15.79:0/1578397415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6c5807be10 0x7f6c5819b9f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 >> 172.21.15.79:0/1578397415 conn(0x7f6c581011c0 msgr2=0x7f6c58101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 shutdown_connections 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:07.119+0000 7f6c5e833640 1 -- 172.21.15.79:0/1578397415 wait complete. 2024-06-01T00:46:08.639 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying mgr service with default placement... 2024-06-01T00:46:09.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:09 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:09.225 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:09 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:09.855 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-01T00:46:09.856 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.022+0000 7fb426605640 1 Processor -- start 2024-06-01T00:46:09.856 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.022+0000 7fb426605640 1 -- start start 2024-06-01T00:46:09.856 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.022+0000 7fb426605640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb420104fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:09.856 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.022+0000 7fb426605640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb420105570 con 0x7fb420104b80 2024-06-01T00:46:09.856 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.022+0000 7fb41ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb420104fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:09.856 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.023+0000 7fb41ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb420104fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56896/0 (socket says 172.21.15.79:56896) 2024-06-01T00:46:09.856 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.023+0000 7fb41ffff640 1 -- 172.21.15.79:0/2781294173 learned_addr learned my addr 172.21.15.79:0/2781294173 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.023+0000 7fb41ffff640 1 -- 172.21.15.79:0/2781294173 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420105d80 con 0x7fb420104b80 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.023+0000 7fb41ffff640 1 --2- 172.21.15.79:0/2781294173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb420104fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb410009820 tx=0x7fb41002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc19951136b07dd1 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb41effd640 1 -- 172.21.15.79:0/2781294173 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb41002ad40 con 0x7fb420104b80 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb41effd640 1 -- 172.21.15.79:0/2781294173 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb410032900 con 0x7fb420104b80 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb41effd640 1 -- 172.21.15.79:0/2781294173 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb410030860 con 0x7fb420104b80 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb426605640 1 -- 172.21.15.79:0/2781294173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 msgr2=0x7fb420104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb426605640 1 --2- 172.21.15.79:0/2781294173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb420104fa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb410009820 tx=0x7fb41002a270 comp rx=0 tx=0).stop 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb426605640 1 -- 172.21.15.79:0/2781294173 shutdown_connections 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb426605640 1 --2- 172.21.15.79:0/2781294173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb420104fa0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb426605640 1 -- 172.21.15.79:0/2781294173 >> 172.21.15.79:0/2781294173 conn(0x7fb4201000e0 msgr2=0x7fb420102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb426605640 1 -- 172.21.15.79:0/2781294173 shutdown_connections 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.024+0000 7fb426605640 1 -- 172.21.15.79:0/2781294173 wait complete. 2024-06-01T00:46:09.857 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.025+0000 7fb426605640 1 Processor -- start 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.025+0000 7fb426605640 1 -- start start 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.025+0000 7fb426605640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb4201987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.025+0000 7fb426605640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb420198d30 con 0x7fb420104b80 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.025+0000 7fb41ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb4201987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.025+0000 7fb41ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb4201987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56906/0 (socket says 172.21.15.79:56906) 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.025+0000 7fb41ffff640 1 -- 172.21.15.79:0/3621542627 learned_addr learned my addr 172.21.15.79:0/3621542627 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.026+0000 7fb41ffff640 1 -- 172.21.15.79:0/3621542627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4100094d0 con 0x7fb420104b80 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.026+0000 7fb41ffff640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb4201987f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb410009940 tx=0x7fb410032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.026+0000 7fb41d7fa640 1 -- 172.21.15.79:0/3621542627 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb410032c80 con 0x7fb420104b80 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.026+0000 7fb41d7fa640 1 -- 172.21.15.79:0/3621542627 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb410032de0 con 0x7fb420104b80 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.026+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb420198f30 con 0x7fb420104b80 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.026+0000 7fb41d7fa640 1 -- 172.21.15.79:0/3621542627 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb41003a730 con 0x7fb420104b80 2024-06-01T00:46:09.858 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.026+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4201993d0 con 0x7fb420104b80 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.027+0000 7fb41d7fa640 1 -- 172.21.15.79:0/3621542627 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fb41003c400 con 0x7fb420104b80 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.027+0000 7fb41d7fa640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb408037860 0x7fb408039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.027+0000 7fb41d7fa640 1 -- 172.21.15.79:0/3621542627 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb41006a980 con 0x7fb420104b80 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.027+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb420108fb0 con 0x7fb420104b80 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.027+0000 7fb41f7fe640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb408037860 0x7fb408039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.027+0000 7fb41f7fe640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb408037860 0x7fb408039d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb4140098c0 tx=0x7fb414006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.032+0000 7fb41d7fa640 1 -- 172.21.15.79:0/3621542627 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb410046460 con 0x7fb420104b80 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.223+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fb420101210 con 0x7fb408037860 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.231+0000 7fb41d7fa640 1 -- 172.21.15.79:0/3621542627 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb420101210 con 0x7fb408037860 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb408037860 msgr2=0x7fb408039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb408037860 0x7fb408039d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb4140098c0 tx=0x7fb414006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 msgr2=0x7fb4201987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:09.859 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb4201987f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb410009940 tx=0x7fb410032670 comp rx=0 tx=0).stop 2024-06-01T00:46:09.860 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 shutdown_connections 2024-06-01T00:46:09.860 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb408037860 0x7fb408039d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:09.860 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 --2- 172.21.15.79:0/3621542627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb420104b80 0x7fb4201987f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:09.860 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 >> 172.21.15.79:0/3621542627 conn(0x7fb4201000e0 msgr2=0x7fb420100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:09.860 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 shutdown_connections 2024-06-01T00:46:09.860 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:09.233+0000 7fb426605640 1 -- 172.21.15.79:0/3621542627 wait complete. 2024-06-01T00:46:09.860 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying crash service with default placement... 2024-06-01T00:46:10.491 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:10 smithi079 ceph-mon[29356]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:10.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:10 smithi079 ceph-mon[29356]: Saving service mgr spec with placement count:2 2024-06-01T00:46:10.492 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:10 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:11.783 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:11 smithi079 ceph-mon[29356]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:11.783 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:11 smithi079 ceph-mon[29356]: Saving service crash spec with placement * 2024-06-01T00:46:11.783 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:11 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:11.783 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:11 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:11.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-01T00:46:11.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.277+0000 7f824b2e6640 1 Processor -- start 2024-06-01T00:46:11.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.277+0000 7f824b2e6640 1 -- start start 2024-06-01T00:46:11.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.278+0000 7f824b2e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f8244104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:11.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.278+0000 7f824b2e6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244105500 con 0x7f8244104b10 2024-06-01T00:46:11.793 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.278+0000 7f8249082640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f8244104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.278+0000 7f8249082640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f8244104f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56912/0 (socket says 172.21.15.79:56912) 2024-06-01T00:46:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.278+0000 7f8249082640 1 -- 172.21.15.79:0/2936304724 learned_addr learned my addr 172.21.15.79:0/2936304724 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.278+0000 7f8249082640 1 -- 172.21.15.79:0/2936304724 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8244105d10 con 0x7f8244104b10 2024-06-01T00:46:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f8249082640 1 --2- 172.21.15.79:0/2936304724 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f8244104f30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8240009820 tx=0x7f824002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1986d6ace0d4f0d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f8237fff640 1 -- 172.21.15.79:0/2936304724 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f824002ad40 con 0x7f8244104b10 2024-06-01T00:46:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f8237fff640 1 -- 172.21.15.79:0/2936304724 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8240032900 con 0x7f8244104b10 2024-06-01T00:46:11.794 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f8237fff640 1 -- 172.21.15.79:0/2936304724 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8240030860 con 0x7f8244104b10 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f824b2e6640 1 -- 172.21.15.79:0/2936304724 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 msgr2=0x7f8244104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f824b2e6640 1 --2- 172.21.15.79:0/2936304724 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f8244104f30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8240009820 tx=0x7f824002a270 comp rx=0 tx=0).stop 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f824b2e6640 1 -- 172.21.15.79:0/2936304724 shutdown_connections 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f824b2e6640 1 --2- 172.21.15.79:0/2936304724 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f8244104f30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.279+0000 7f824b2e6640 1 -- 172.21.15.79:0/2936304724 >> 172.21.15.79:0/2936304724 conn(0x7f82441000d0 msgr2=0x7f82441024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f824b2e6640 1 -- 172.21.15.79:0/2936304724 shutdown_connections 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f824b2e6640 1 -- 172.21.15.79:0/2936304724 wait complete. 2024-06-01T00:46:11.795 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f824b2e6640 1 Processor -- start 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f824b2e6640 1 -- start start 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f824b2e6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f82441987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f824b2e6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8244198d00 con 0x7f8244104b10 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f8249082640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f82441987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f8249082640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f82441987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56926/0 (socket says 172.21.15.79:56926) 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.280+0000 7f8249082640 1 -- 172.21.15.79:0/3188508342 learned_addr learned my addr 172.21.15.79:0/3188508342 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.281+0000 7f8249082640 1 -- 172.21.15.79:0/3188508342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82400094d0 con 0x7f8244104b10 2024-06-01T00:46:11.796 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.281+0000 7f8249082640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f82441987c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f82400097f0 tx=0x7f8240030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.281+0000 7f82367fc640 1 -- 172.21.15.79:0/3188508342 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8240040070 con 0x7f8244104b10 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.281+0000 7f82367fc640 1 -- 172.21.15.79:0/3188508342 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8240032dd0 con 0x7f8244104b10 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.281+0000 7f82367fc640 1 -- 172.21.15.79:0/3188508342 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f824003a800 con 0x7f8244104b10 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.281+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8244198f00 con 0x7f8244104b10 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.281+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82441993a0 con 0x7f8244104b10 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.282+0000 7f82367fc640 1 -- 172.21.15.79:0/3188508342 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f824003a960 con 0x7f8244104b10 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.282+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8244108f40 con 0x7f8244104b10 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.282+0000 7f82367fc640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f82300378b0 0x7f8230039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:11.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.282+0000 7f82367fc640 1 -- 172.21.15.79:0/3188508342 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f824006a5e0 con 0x7f8244104b10 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.282+0000 7f8248881640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f82300378b0 0x7f8230039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.282+0000 7f8248881640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f82300378b0 0x7f8230039d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f82380098c0 tx=0x7f8238006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.287+0000 7f82367fc640 1 -- 172.21.15.79:0/3188508342 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8240039310 con 0x7f8244104b10 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.490+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f8244101240 con 0x7f82300378b0 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.497+0000 7f82367fc640 1 -- 172.21.15.79:0/3188508342 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f8244101240 con 0x7f82300378b0 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f82300378b0 msgr2=0x7f8230039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f82300378b0 0x7f8230039d70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f82380098c0 tx=0x7f8238006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:11.798 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 msgr2=0x7f82441987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f82441987c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f82400097f0 tx=0x7f8240030e90 comp rx=0 tx=0).stop 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 shutdown_connections 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f82300378b0 0x7f8230039d70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 --2- 172.21.15.79:0/3188508342 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8244104b10 0x7f82441987c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 >> 172.21.15.79:0/3188508342 conn(0x7f82441000d0 msgr2=0x7f8244100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 shutdown_connections 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:10.499+0000 7f824b2e6640 1 -- 172.21.15.79:0/3188508342 wait complete. 2024-06-01T00:46:11.799 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying ceph-exporter service with default placement... 2024-06-01T00:46:13.008 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-06-01T00:46:13.008 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52ad465640 1 Processor -- start 2024-06-01T00:46:13.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52ad465640 1 -- start start 2024-06-01T00:46:13.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52ad465640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a8106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:13.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52ad465640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8106630 con 0x7f52a8105c40 2024-06-01T00:46:13.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a8106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:13.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a8106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39568/0 (socket says 172.21.15.79:39568) 2024-06-01T00:46:13.009 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52a6ffd640 1 -- 172.21.15.79:0/1946497270 learned_addr learned my addr 172.21.15.79:0/1946497270 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.187+0000 7f52a6ffd640 1 -- 172.21.15.79:0/1946497270 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a8106e40 con 0x7f52a8105c40 2024-06-01T00:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.188+0000 7f52a6ffd640 1 --2- 172.21.15.79:0/1946497270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a8106060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5298009820 tx=0x7f529802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d3ec46ad2dc9a6 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.188+0000 7f52a5ffb640 1 -- 172.21.15.79:0/1946497270 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529802ad40 con 0x7f52a8105c40 2024-06-01T00:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.188+0000 7f52a5ffb640 1 -- 172.21.15.79:0/1946497270 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5298032900 con 0x7f52a8105c40 2024-06-01T00:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.188+0000 7f52a5ffb640 1 -- 172.21.15.79:0/1946497270 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5298030860 con 0x7f52a8105c40 2024-06-01T00:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 -- 172.21.15.79:0/1946497270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 msgr2=0x7f52a8106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:13.010 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 --2- 172.21.15.79:0/1946497270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a8106060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5298009820 tx=0x7f529802a270 comp rx=0 tx=0).stop 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 -- 172.21.15.79:0/1946497270 shutdown_connections 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 --2- 172.21.15.79:0/1946497270 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a8106060 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 -- 172.21.15.79:0/1946497270 >> 172.21.15.79:0/1946497270 conn(0x7f52a81011c0 msgr2=0x7f52a8103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 -- 172.21.15.79:0/1946497270 shutdown_connections 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 -- 172.21.15.79:0/1946497270 wait complete. 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 Processor -- start 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.189+0000 7f52ad465640 1 -- start start 2024-06-01T00:46:13.011 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52ad465640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a81998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52ad465640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8199de0 con 0x7f52a8105c40 2024-06-01T00:46:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a81998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52a6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a81998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39574/0 (socket says 172.21.15.79:39574) 2024-06-01T00:46:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52a6ffd640 1 -- 172.21.15.79:0/3255634071 learned_addr learned my addr 172.21.15.79:0/3255634071 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52a6ffd640 1 -- 172.21.15.79:0/3255634071 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980094d0 con 0x7f52a8105c40 2024-06-01T00:46:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52a6ffd640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a81998a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5298009940 tx=0x7f5298032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:13.012 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f528ffff640 1 -- 172.21.15.79:0/3255634071 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5298032c80 con 0x7f52a8105c40 2024-06-01T00:46:13.013 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f528ffff640 1 -- 172.21.15.79:0/3255634071 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5298032de0 con 0x7f52a8105c40 2024-06-01T00:46:13.013 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a8199fe0 con 0x7f52a8105c40 2024-06-01T00:46:13.013 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f528ffff640 1 -- 172.21.15.79:0/3255634071 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f529803a730 con 0x7f52a8105c40 2024-06-01T00:46:13.013 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.190+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a819a480 con 0x7f52a8105c40 2024-06-01T00:46:13.013 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.191+0000 7f528ffff640 1 -- 172.21.15.79:0/3255634071 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f529803a9f0 con 0x7f52a8105c40 2024-06-01T00:46:13.013 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.191+0000 7f528ffff640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5290037860 0x7f5290039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:13.014 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.191+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a810a070 con 0x7f52a8105c40 2024-06-01T00:46:13.014 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.191+0000 7f528ffff640 1 -- 172.21.15.79:0/3255634071 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f529806a8e0 con 0x7f52a8105c40 2024-06-01T00:46:13.014 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.191+0000 7f52a67fc640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5290037860 0x7f5290039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:13.014 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.192+0000 7f52a67fc640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5290037860 0x7f5290039d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f529c0098c0 tx=0x7f529c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:13.014 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.196+0000 7f528ffff640 1 -- 172.21.15.79:0/3255634071 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5298039b90 con 0x7f52a8105c40 2024-06-01T00:46:13.015 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.387+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f52a81022e0 con 0x7f5290037860 2024-06-01T00:46:13.015 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.403+0000 7f528ffff640 1 -- 172.21.15.79:0/3255634071 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f52a81022e0 con 0x7f5290037860 2024-06-01T00:46:13.015 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5290037860 msgr2=0x7f5290039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:13.015 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5290037860 0x7f5290039d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f529c0098c0 tx=0x7f529c006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:13.015 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 msgr2=0x7f52a81998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:13.015 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a81998a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5298009940 tx=0x7f5298032670 comp rx=0 tx=0).stop 2024-06-01T00:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 shutdown_connections 2024-06-01T00:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5290037860 0x7f5290039d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 --2- 172.21.15.79:0/3255634071 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f52a8105c40 0x7f52a81998a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 >> 172.21.15.79:0/3255634071 conn(0x7f52a81011c0 msgr2=0x7f52a8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 shutdown_connections 2024-06-01T00:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:12.405+0000 7f52ad465640 1 -- 172.21.15.79:0/3255634071 wait complete. 2024-06-01T00:46:13.016 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying prometheus service with default placement... 2024-06-01T00:46:13.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:13 smithi079 ceph-mon[29356]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:13.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:13 smithi079 ceph-mon[29356]: Saving service ceph-exporter spec with placement * 2024-06-01T00:46:13.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:13 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:13.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:13 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:14.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:14 smithi079 ceph-mon[29356]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:14.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:14 smithi079 ceph-mon[29356]: Saving service prometheus spec with placement count:1 2024-06-01T00:46:14.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:14 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:15.043 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-01T00:46:15.043 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.378+0000 7f6b25f4d640 1 Processor -- start 2024-06-01T00:46:15.043 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.379+0000 7f6b25f4d640 1 -- start start 2024-06-01T00:46:15.043 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.379+0000 7f6b25f4d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b20106040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:15.043 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.379+0000 7f6b25f4d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b20106610 con 0x7f6b20105c20 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.379+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b20106040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.379+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b20106040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39580/0 (socket says 172.21.15.79:39580) 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.379+0000 7f6b1f7fe640 1 -- 172.21.15.79:0/4174665912 learned_addr learned my addr 172.21.15.79:0/4174665912 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.379+0000 7f6b1f7fe640 1 -- 172.21.15.79:0/4174665912 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20106e20 con 0x7f6b20105c20 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.380+0000 7f6b1f7fe640 1 --2- 172.21.15.79:0/4174665912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b20106040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009820 tx=0x7f6b1002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674911d6bf419db1 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.380+0000 7f6b1effd640 1 -- 172.21.15.79:0/4174665912 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b1002ad40 con 0x7f6b20105c20 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.380+0000 7f6b1effd640 1 -- 172.21.15.79:0/4174665912 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b10032900 con 0x7f6b20105c20 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.380+0000 7f6b1effd640 1 -- 172.21.15.79:0/4174665912 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b10030860 con 0x7f6b20105c20 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 -- 172.21.15.79:0/4174665912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 msgr2=0x7f6b20106040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 --2- 172.21.15.79:0/4174665912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b20106040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009820 tx=0x7f6b1002a270 comp rx=0 tx=0).stop 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 -- 172.21.15.79:0/4174665912 shutdown_connections 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 --2- 172.21.15.79:0/4174665912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b20106040 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:15.044 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 -- 172.21.15.79:0/4174665912 >> 172.21.15.79:0/4174665912 conn(0x7f6b201011a0 msgr2=0x7f6b20103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 -- 172.21.15.79:0/4174665912 shutdown_connections 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 -- 172.21.15.79:0/4174665912 wait complete. 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 Processor -- start 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.381+0000 7f6b25f4d640 1 -- start start 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b25f4d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b201998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b25f4d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b20199e10 con 0x7f6b20105c20 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b201998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b201998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39586/0 (socket says 172.21.15.79:39586) 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1f7fe640 1 -- 172.21.15.79:0/915082221 learned_addr learned my addr 172.21.15.79:0/915082221 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1f7fe640 1 -- 172.21.15.79:0/915082221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b100094d0 con 0x7f6b20105c20 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1f7fe640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b201998d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b100097f0 tx=0x7f6b10030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1d7fa640 1 -- 172.21.15.79:0/915082221 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b10040070 con 0x7f6b20105c20 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1d7fa640 1 -- 172.21.15.79:0/915082221 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b10032dd0 con 0x7f6b20105c20 2024-06-01T00:46:15.045 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2019a010 con 0x7f6b20105c20 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b1d7fa640 1 -- 172.21.15.79:0/915082221 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b1003a800 con 0x7f6b20105c20 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.382+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2019a4b0 con 0x7f6b20105c20 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.383+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b2010a050 con 0x7f6b20105c20 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.384+0000 7f6b1d7fa640 1 -- 172.21.15.79:0/915082221 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f6b1003ccc0 con 0x7f6b20105c20 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.384+0000 7f6b1d7fa640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6b0c0378b0 0x7f6b0c039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.384+0000 7f6b1d7fa640 1 -- 172.21.15.79:0/915082221 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6b1006a4e0 con 0x7f6b20105c20 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.384+0000 7f6b16ffd640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6b0c0378b0 0x7f6b0c039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.384+0000 7f6b16ffd640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6b0c0378b0 0x7f6b0c039d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6b080098c0 tx=0x7f6b08006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.388+0000 7f6b1d7fa640 1 -- 172.21.15.79:0/915082221 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6b10043920 con 0x7f6b20105c20 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.584+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b20102340 con 0x7f6b0c0378b0 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.589+0000 7f6b1d7fa640 1 -- 172.21.15.79:0/915082221 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f6b20102340 con 0x7f6b0c0378b0 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.590+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6b0c0378b0 msgr2=0x7f6b0c039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.590+0000 7f6b25f4d640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6b0c0378b0 0x7f6b0c039d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6b080098c0 tx=0x7f6b08006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:15.046 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.590+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 msgr2=0x7f6b201998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.590+0000 7f6b25f4d640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b201998d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b100097f0 tx=0x7f6b10030e90 comp rx=0 tx=0).stop 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.591+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 shutdown_connections 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.591+0000 7f6b25f4d640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f6b0c0378b0 0x7f6b0c039d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.591+0000 7f6b25f4d640 1 --2- 172.21.15.79:0/915082221 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b20105c20 0x7f6b201998d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.591+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 >> 172.21.15.79:0/915082221 conn(0x7f6b201011a0 msgr2=0x7f6b20101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.591+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 shutdown_connections 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:13.591+0000 7f6b25f4d640 1 -- 172.21.15.79:0/915082221 wait complete. 2024-06-01T00:46:15.047 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying grafana service with default placement... 2024-06-01T00:46:16.338 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-01T00:46:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.451+0000 7f4fc802f640 1 Processor -- start 2024-06-01T00:46:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.451+0000 7f4fc802f640 1 -- start start 2024-06-01T00:46:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.451+0000 7f4fc802f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0108ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.451+0000 7f4fc802f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc0079e00 con 0x7f4fc01066c0 2024-06-01T00:46:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.452+0000 7f4fc5dcb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0108ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:16.339 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.452+0000 7f4fc5dcb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0108ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39594/0 (socket says 172.21.15.79:39594) 2024-06-01T00:46:16.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.452+0000 7f4fc5dcb640 1 -- 172.21.15.79:0/3832022525 learned_addr learned my addr 172.21.15.79:0/3832022525 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:16.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.452+0000 7f4fc5dcb640 1 -- 172.21.15.79:0/3832022525 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0108ff0 con 0x7f4fc01066c0 2024-06-01T00:46:16.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc5dcb640 1 --2- 172.21.15.79:0/3832022525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0108ab0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009820 tx=0x7f4fbc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e355c0d6988db39 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:16.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc4dc9640 1 -- 172.21.15.79:0/3832022525 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc02ad40 con 0x7f4fc01066c0 2024-06-01T00:46:16.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc4dc9640 1 -- 172.21.15.79:0/3832022525 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fbc032900 con 0x7f4fc01066c0 2024-06-01T00:46:16.340 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc4dc9640 1 -- 172.21.15.79:0/3832022525 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc030860 con 0x7f4fc01066c0 2024-06-01T00:46:16.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc802f640 1 -- 172.21.15.79:0/3832022525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 msgr2=0x7f4fc0108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:16.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc802f640 1 --2- 172.21.15.79:0/3832022525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0108ab0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009820 tx=0x7f4fbc02a270 comp rx=0 tx=0).stop 2024-06-01T00:46:16.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc802f640 1 -- 172.21.15.79:0/3832022525 shutdown_connections 2024-06-01T00:46:16.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc802f640 1 --2- 172.21.15.79:0/3832022525 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0108ab0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:16.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.453+0000 7f4fc802f640 1 -- 172.21.15.79:0/3832022525 >> 172.21.15.79:0/3832022525 conn(0x7f4fc01000d0 msgr2=0x7f4fc01024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:16.341 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc802f640 1 -- 172.21.15.79:0/3832022525 shutdown_connections 2024-06-01T00:46:16.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc802f640 1 -- 172.21.15.79:0/3832022525 wait complete. 2024-06-01T00:46:16.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc802f640 1 Processor -- start 2024-06-01T00:46:16.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc802f640 1 -- start start 2024-06-01T00:46:16.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc802f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0105ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:16.342 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc802f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc0106430 con 0x7f4fc01066c0 2024-06-01T00:46:16.344 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc5dcb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0105ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:16.344 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc5dcb640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0105ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39606/0 (socket says 172.21.15.79:39606) 2024-06-01T00:46:16.344 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.454+0000 7f4fc5dcb640 1 -- 172.21.15.79:0/847040147 learned_addr learned my addr 172.21.15.79:0/847040147 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:16.344 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.455+0000 7f4fc5dcb640 1 -- 172.21.15.79:0/847040147 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc0094d0 con 0x7f4fc01066c0 2024-06-01T00:46:16.345 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.455+0000 7f4fc5dcb640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0105ef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0097f0 tx=0x7f4fbc030e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:16.345 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.455+0000 7f4fb2ffd640 1 -- 172.21.15.79:0/847040147 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc032c80 con 0x7f4fc01066c0 2024-06-01T00:46:16.345 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.455+0000 7f4fb2ffd640 1 -- 172.21.15.79:0/847040147 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fbc032de0 con 0x7f4fc01066c0 2024-06-01T00:46:16.345 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.455+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc0104670 con 0x7f4fc01066c0 2024-06-01T00:46:16.345 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.455+0000 7f4fb2ffd640 1 -- 172.21.15.79:0/847040147 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc03a7c0 con 0x7f4fc01066c0 2024-06-01T00:46:16.345 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.455+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc0104b10 con 0x7f4fc01066c0 2024-06-01T00:46:16.346 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.456+0000 7f4fb2ffd640 1 -- 172.21.15.79:0/847040147 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f4fbc03a9c0 con 0x7f4fc01066c0 2024-06-01T00:46:16.346 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.456+0000 7f4fb2ffd640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4fac037860 0x7f4fac039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:16.346 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.456+0000 7f4fb2ffd640 1 -- 172.21.15.79:0/847040147 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4fbc06b8e0 con 0x7f4fc01066c0 2024-06-01T00:46:16.346 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.456+0000 7f4fc55ca640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4fac037860 0x7f4fac039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:16.346 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.456+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc0061ae0 con 0x7f4fc01066c0 2024-06-01T00:46:16.346 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.457+0000 7f4fc55ca640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4fac037860 0x7f4fac039d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40098c0 tx=0x7f4fb4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:16.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.461+0000 7f4fb2ffd640 1 -- 172.21.15.79:0/847040147 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4fbc039d50 con 0x7f4fc01066c0 2024-06-01T00:46:16.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.651+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fc0104fb0 con 0x7f4fac037860 2024-06-01T00:46:16.347 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.655+0000 7f4fb2ffd640 1 -- 172.21.15.79:0/847040147 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f4fc0104fb0 con 0x7f4fac037860 2024-06-01T00:46:16.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4fac037860 msgr2=0x7f4fac039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:16.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4fac037860 0x7f4fac039d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40098c0 tx=0x7f4fb4006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:16.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 msgr2=0x7f4fc0105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:16.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0105ef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0097f0 tx=0x7f4fbc030e20 comp rx=0 tx=0).stop 2024-06-01T00:46:16.348 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 shutdown_connections 2024-06-01T00:46:16.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4fac037860 0x7f4fac039d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:16.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 --2- 172.21.15.79:0/847040147 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4fc01066c0 0x7f4fc0105ef0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:16.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.657+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 >> 172.21.15.79:0/847040147 conn(0x7f4fc01000d0 msgr2=0x7f4fc0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:16.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.658+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 shutdown_connections 2024-06-01T00:46:16.349 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:15.658+0000 7f4fc802f640 1 -- 172.21.15.79:0/847040147 wait complete. 2024-06-01T00:46:16.349 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying node-exporter service with default placement... 2024-06-01T00:46:16.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:16 smithi079 ceph-mon[29356]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:16.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:16 smithi079 ceph-mon[29356]: Saving service grafana spec with placement count:1 2024-06-01T00:46:16.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:16 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:16.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:16 smithi079 ceph-mon[29356]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:17.406 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-01T00:46:17.406 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.780+0000 7f4d682d4640 1 Processor -- start 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.780+0000 7f4d682d4640 1 -- start start 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.781+0000 7f4d682d4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d60108ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.781+0000 7f4d682d4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60079de0 con 0x7f4d601066e0 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.781+0000 7f4d66070640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d60108ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.781+0000 7f4d66070640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d60108ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39612/0 (socket says 172.21.15.79:39612) 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.781+0000 7f4d66070640 1 -- 172.21.15.79:0/3371868123 learned_addr learned my addr 172.21.15.79:0/3371868123 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.781+0000 7f4d66070640 1 -- 172.21.15.79:0/3371868123 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60109010 con 0x7f4d601066e0 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.782+0000 7f4d66070640 1 --2- 172.21.15.79:0/3371868123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d60108ad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009820 tx=0x7f4d5c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25607dc6ae419884 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.782+0000 7f4d6506e640 1 -- 172.21.15.79:0/3371868123 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d5c02ad40 con 0x7f4d601066e0 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.782+0000 7f4d6506e640 1 -- 172.21.15.79:0/3371868123 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d5c032900 con 0x7f4d601066e0 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.782+0000 7f4d6506e640 1 -- 172.21.15.79:0/3371868123 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d5c030860 con 0x7f4d601066e0 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 -- 172.21.15.79:0/3371868123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 msgr2=0x7f4d60108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:17.407 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 --2- 172.21.15.79:0/3371868123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d60108ad0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009820 tx=0x7f4d5c02a270 comp rx=0 tx=0).stop 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 -- 172.21.15.79:0/3371868123 shutdown_connections 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 --2- 172.21.15.79:0/3371868123 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d60108ad0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 -- 172.21.15.79:0/3371868123 >> 172.21.15.79:0/3371868123 conn(0x7f4d601000d0 msgr2=0x7f4d601024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 -- 172.21.15.79:0/3371868123 shutdown_connections 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 -- 172.21.15.79:0/3371868123 wait complete. 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 Processor -- start 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.783+0000 7f4d682d4640 1 -- start start 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d682d4640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d601998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d682d4640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60199de0 con 0x7f4d601066e0 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d66070640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d601998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d66070640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d601998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39620/0 (socket says 172.21.15.79:39620) 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d66070640 1 -- 172.21.15.79:0/4184214692 learned_addr learned my addr 172.21.15.79:0/4184214692 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:17.408 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d66070640 1 -- 172.21.15.79:0/4184214692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c0094d0 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d66070640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d601998a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0097f0 tx=0x7f4d5c030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d537fe640 1 -- 172.21.15.79:0/4184214692 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d5c040070 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d60199fe0 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d537fe640 1 -- 172.21.15.79:0/4184214692 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d5c032dd0 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d537fe640 1 -- 172.21.15.79:0/4184214692 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d5c03a800 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.784+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6019a480 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.785+0000 7f4d537fe640 1 -- 172.21.15.79:0/4184214692 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f4d5c03c400 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.785+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60109e10 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.785+0000 7f4d537fe640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4d4c037860 0x7f4d4c039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.785+0000 7f4d537fe640 1 -- 172.21.15.79:0/4184214692 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4d5c06a530 con 0x7f4d601066e0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.786+0000 7f4d6586f640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4d4c037860 0x7f4d4c039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.786+0000 7f4d6586f640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4d4c037860 0x7f4d4c039d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d540098c0 tx=0x7f4d54006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:17.409 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.790+0000 7f4d537fe640 1 -- 172.21.15.79:0/4184214692 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4d5c043ac0 con 0x7f4d601066e0 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.974+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d60101240 con 0x7f4d4c037860 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.979+0000 7f4d537fe640 1 -- 172.21.15.79:0/4184214692 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f4d60101240 con 0x7f4d4c037860 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.981+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4d4c037860 msgr2=0x7f4d4c039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.981+0000 7f4d682d4640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4d4c037860 0x7f4d4c039d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d540098c0 tx=0x7f4d54006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.981+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 msgr2=0x7f4d601998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.981+0000 7f4d682d4640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d601998a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c0097f0 tx=0x7f4d5c030e90 comp rx=0 tx=0).stop 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.982+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 shutdown_connections 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.982+0000 7f4d682d4640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f4d4c037860 0x7f4d4c039d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.982+0000 7f4d682d4640 1 --2- 172.21.15.79:0/4184214692 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d601066e0 0x7f4d601998a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.982+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 >> 172.21.15.79:0/4184214692 conn(0x7f4d601000d0 msgr2=0x7f4d60100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.982+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 shutdown_connections 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:16.982+0000 7f4d682d4640 1 -- 172.21.15.79:0/4184214692 wait complete. 2024-06-01T00:46:17.410 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying alertmanager service with default placement... 2024-06-01T00:46:18.015 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:17 smithi079 ceph-mon[29356]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:18.016 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:17 smithi079 ceph-mon[29356]: Saving service node-exporter spec with placement * 2024-06-01T00:46:18.016 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:17 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:18.579 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-01T00:46:18.579 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.820+0000 7fed47063640 1 Processor -- start 2024-06-01T00:46:18.579 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.820+0000 7fed47063640 1 -- start start 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.820+0000 7fed47063640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed40106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.820+0000 7fed47063640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed40106630 con 0x7fed40105c40 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.821+0000 7fed44dff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed40106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.821+0000 7fed44dff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed40106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39622/0 (socket says 172.21.15.79:39622) 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.821+0000 7fed44dff640 1 -- 172.21.15.79:0/176150709 learned_addr learned my addr 172.21.15.79:0/176150709 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.821+0000 7fed44dff640 1 -- 172.21.15.79:0/176150709 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed40106e40 con 0x7fed40105c40 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.822+0000 7fed44dff640 1 --2- 172.21.15.79:0/176150709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed40106060 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed34009820 tx=0x7fed3402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52032cdc54e127f6 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.822+0000 7fed3f7fe640 1 -- 172.21.15.79:0/176150709 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3402ad40 con 0x7fed40105c40 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.822+0000 7fed3f7fe640 1 -- 172.21.15.79:0/176150709 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed34032900 con 0x7fed40105c40 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.822+0000 7fed3f7fe640 1 -- 172.21.15.79:0/176150709 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed34030860 con 0x7fed40105c40 2024-06-01T00:46:18.580 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.822+0000 7fed47063640 1 -- 172.21.15.79:0/176150709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 msgr2=0x7fed40106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.822+0000 7fed47063640 1 --2- 172.21.15.79:0/176150709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed40106060 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fed34009820 tx=0x7fed3402a270 comp rx=0 tx=0).stop 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 -- 172.21.15.79:0/176150709 shutdown_connections 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 --2- 172.21.15.79:0/176150709 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed40106060 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 -- 172.21.15.79:0/176150709 >> 172.21.15.79:0/176150709 conn(0x7fed401011c0 msgr2=0x7fed40103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 -- 172.21.15.79:0/176150709 shutdown_connections 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 -- 172.21.15.79:0/176150709 wait complete. 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 Processor -- start 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 -- start start 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed401998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed47063640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed40199de0 con 0x7fed40105c40 2024-06-01T00:46:18.581 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed44dff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed401998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed44dff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed401998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39636/0 (socket says 172.21.15.79:39636) 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.823+0000 7fed44dff640 1 -- 172.21.15.79:0/1492923357 learned_addr learned my addr 172.21.15.79:0/1492923357 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.824+0000 7fed44dff640 1 -- 172.21.15.79:0/1492923357 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed340094d0 con 0x7fed40105c40 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.824+0000 7fed44dff640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed401998a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fed340097f0 tx=0x7fed34030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.824+0000 7fed3dffb640 1 -- 172.21.15.79:0/1492923357 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed34040070 con 0x7fed40105c40 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.824+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed40199fe0 con 0x7fed40105c40 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.824+0000 7fed3dffb640 1 -- 172.21.15.79:0/1492923357 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed34032dd0 con 0x7fed40105c40 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.824+0000 7fed3dffb640 1 -- 172.21.15.79:0/1492923357 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed3403a800 con 0x7fed40105c40 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.824+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4019a480 con 0x7fed40105c40 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.825+0000 7fed3dffb640 1 -- 172.21.15.79:0/1492923357 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fed3403a960 con 0x7fed40105c40 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.825+0000 7fed3dffb640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fed30037860 0x7fed30039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:18.582 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.825+0000 7fed3dffb640 1 -- 172.21.15.79:0/1492923357 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fed3406a540 con 0x7fed40105c40 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.825+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4010a070 con 0x7fed40105c40 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.825+0000 7fed3ffff640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fed30037860 0x7fed30039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.826+0000 7fed3ffff640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fed30037860 0x7fed30039d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0098c0 tx=0x7fed2c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:17.830+0000 7fed3dffb640 1 -- 172.21.15.79:0/1492923357 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fed34043920 con 0x7fed40105c40 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.013+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fed401022d0 con 0x7fed30037860 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.018+0000 7fed3dffb640 1 -- 172.21.15.79:0/1492923357 <== mgr.14118 v2:172.21.15.79:6800/1904937508 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fed401022d0 con 0x7fed30037860 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fed30037860 msgr2=0x7fed30039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fed30037860 0x7fed30039d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed2c0098c0 tx=0x7fed2c006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 msgr2=0x7fed401998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed401998a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fed340097f0 tx=0x7fed34030e90 comp rx=0 tx=0).stop 2024-06-01T00:46:18.583 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 shutdown_connections 2024-06-01T00:46:18.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fed30037860 0x7fed30039d20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:18.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 --2- 172.21.15.79:0/1492923357 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fed40105c40 0x7fed401998a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:18.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 >> 172.21.15.79:0/1492923357 conn(0x7fed401011c0 msgr2=0x7fed40101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:18.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 shutdown_connections 2024-06-01T00:46:18.584 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.020+0000 7fed47063640 1 -- 172.21.15.79:0/1492923357 wait complete. 2024-06-01T00:46:19.158 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:19 smithi079 ceph-mon[29356]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:19.158 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:19 smithi079 ceph-mon[29356]: Saving service alertmanager spec with placement count:1 2024-06-01T00:46:19.158 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:19 smithi079 ceph-mon[29356]: from='mgr.14118 172.21.15.79:0/1136730099' entity='mgr.smithi079.emylav' 2024-06-01T00:46:19.158 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:19 smithi079 ceph-mon[29356]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:19.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fd561e640 1 Processor -- start 2024-06-01T00:46:19.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fd561e640 1 -- start start 2024-06-01T00:46:19.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fd561e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0108b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:19.768 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fd561e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0079da0 con 0x7f5fd0106790 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0108b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0108b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39644/0 (socket says 172.21.15.79:39644) 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fceffd640 1 -- 172.21.15.79:0/564790394 learned_addr learned my addr 172.21.15.79:0/564790394 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.971+0000 7f5fceffd640 1 -- 172.21.15.79:0/564790394 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0079ee0 con 0x7f5fd0106790 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.972+0000 7f5fceffd640 1 --2- 172.21.15.79:0/564790394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0108b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009820 tx=0x7f5fc002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd1703bdefa8e3a server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fce7fc640 1 -- 172.21.15.79:0/564790394 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fc002ad40 con 0x7f5fd0106790 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fce7fc640 1 -- 172.21.15.79:0/564790394 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fc0032900 con 0x7f5fd0106790 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fce7fc640 1 -- 172.21.15.79:0/564790394 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fc0030860 con 0x7f5fd0106790 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fd561e640 1 -- 172.21.15.79:0/564790394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 msgr2=0x7f5fd0108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fd561e640 1 --2- 172.21.15.79:0/564790394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0108b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009820 tx=0x7f5fc002a270 comp rx=0 tx=0).stop 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fd561e640 1 -- 172.21.15.79:0/564790394 shutdown_connections 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fd561e640 1 --2- 172.21.15.79:0/564790394 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0108b80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:19.769 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fd561e640 1 -- 172.21.15.79:0/564790394 >> 172.21.15.79:0/564790394 conn(0x7f5fd01000e0 msgr2=0x7f5fd0102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fd561e640 1 -- 172.21.15.79:0/564790394 shutdown_connections 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.973+0000 7f5fd561e640 1 -- 172.21.15.79:0/564790394 wait complete. 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fd561e640 1 Processor -- start 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fd561e640 1 -- start start 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fd561e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fd561e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0199db0 con 0x7f5fd0106790 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fceffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39656/0 (socket says 172.21.15.79:39656) 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fceffd640 1 -- 172.21.15.79:0/1831810319 learned_addr learned my addr 172.21.15.79:0/1831810319 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.974+0000 7f5fceffd640 1 -- 172.21.15.79:0/1831810319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc00094d0 con 0x7f5fd0106790 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.975+0000 7f5fceffd640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0199870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00097f0 tx=0x7f5fc0030e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:19.770 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.975+0000 7f5fccff9640 1 -- 172.21.15.79:0/1831810319 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fc0032c80 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.975+0000 7f5fccff9640 1 -- 172.21.15.79:0/1831810319 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fc0032de0 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.975+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd0199fb0 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.975+0000 7f5fccff9640 1 -- 172.21.15.79:0/1831810319 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fc003a7c0 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.975+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd019a450 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.976+0000 7f5fccff9640 1 -- 172.21.15.79:0/1831810319 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f5fc003a9c0 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.976+0000 7f5fccff9640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5fbc037860 0x7f5fbc039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.976+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd0109e50 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.976+0000 7f5fccff9640 1 -- 172.21.15.79:0/1831810319 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5fc006aa80 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.976+0000 7f5fc67fc640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5fbc037860 0x7f5fbc039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.977+0000 7f5fc67fc640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5fbc037860 0x7f5fbc039d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80098c0 tx=0x7f5fb8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:18.981+0000 7f5fccff9640 1 -- 172.21.15.79:0/1831810319 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5fc009a900 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.156+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5fd010a090 con 0x7f5fd0106790 2024-06-01T00:46:19.771 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.159+0000 7f5fccff9640 1 -- 172.21.15.79:0/1831810319 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5fc0043310 con 0x7f5fd0106790 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5fbc037860 msgr2=0x7f5fbc039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5fbc037860 0x7f5fbc039d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80098c0 tx=0x7f5fb8006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 msgr2=0x7f5fd0199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0199870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00097f0 tx=0x7f5fc0030e20 comp rx=0 tx=0).stop 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 shutdown_connections 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5fbc037860 0x7f5fbc039d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 --2- 172.21.15.79:0/1831810319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5fd0106790 0x7f5fd0199870 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 >> 172.21.15.79:0/1831810319 conn(0x7f5fd01000e0 msgr2=0x7f5fd0100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 shutdown_connections 2024-06-01T00:46:19.772 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:19.162+0000 7f5fd561e640 1 -- 172.21.15.79:0/1831810319 wait complete. 2024-06-01T00:46:20.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:20 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1831810319' entity='client.admin' 2024-06-01T00:46:20.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.151+0000 7fb8c5f60640 1 Processor -- start 2024-06-01T00:46:20.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.151+0000 7fb8c5f60640 1 -- start start 2024-06-01T00:46:20.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.151+0000 7fb8c5f60640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.151+0000 7fb8c5f60640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c0106630 con 0x7fb8c0105c40 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.151+0000 7fb8bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.151+0000 7fb8bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39658/0 (socket says 172.21.15.79:39658) 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.151+0000 7fb8bf7fe640 1 -- 172.21.15.79:0/4009437821 learned_addr learned my addr 172.21.15.79:0/4009437821 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.152+0000 7fb8bf7fe640 1 -- 172.21.15.79:0/4009437821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c0106e40 con 0x7fb8c0105c40 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.152+0000 7fb8bf7fe640 1 --2- 172.21.15.79:0/4009437821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0106060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009e80 tx=0x7fb8b002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2f4b9ed760f9d9 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.152+0000 7fb8beffd640 1 -- 172.21.15.79:0/4009437821 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b0031400 con 0x7fb8c0105c40 2024-06-01T00:46:20.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.152+0000 7fb8beffd640 1 -- 172.21.15.79:0/4009437821 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8b002d050 con 0x7fb8c0105c40 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.152+0000 7fb8beffd640 1 -- 172.21.15.79:0/4009437821 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b0007930 con 0x7fb8c0105c40 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.153+0000 7fb8c5f60640 1 -- 172.21.15.79:0/4009437821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 msgr2=0x7fb8c0106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.153+0000 7fb8c5f60640 1 --2- 172.21.15.79:0/4009437821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0106060 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009e80 tx=0x7fb8b002a7a0 comp rx=0 tx=0).stop 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.153+0000 7fb8c5f60640 1 -- 172.21.15.79:0/4009437821 shutdown_connections 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.153+0000 7fb8c5f60640 1 --2- 172.21.15.79:0/4009437821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0106060 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.153+0000 7fb8c5f60640 1 -- 172.21.15.79:0/4009437821 >> 172.21.15.79:0/4009437821 conn(0x7fb8c01011c0 msgr2=0x7fb8c0103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.153+0000 7fb8c5f60640 1 -- 172.21.15.79:0/4009437821 shutdown_connections 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.153+0000 7fb8c5f60640 1 -- 172.21.15.79:0/4009437821 wait complete. 2024-06-01T00:46:20.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8c5f60640 1 Processor -- start 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8c5f60640 1 -- start start 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8c5f60640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8c5f60640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c0199e80 con 0x7fb8c0105c40 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8bf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39670/0 (socket says 172.21.15.79:39670) 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8bf7fe640 1 -- 172.21.15.79:0/624576285 learned_addr learned my addr 172.21.15.79:0/624576285 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8bf7fe640 1 -- 172.21.15.79:0/624576285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b0009b30 con 0x7fb8c0105c40 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.154+0000 7fb8bf7fe640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0199940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8b002acc0 tx=0x7fb8b0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.155+0000 7fb8bd7fa640 1 -- 172.21.15.79:0/624576285 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b003a550 con 0x7fb8c0105c40 2024-06-01T00:46:20.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.155+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c019a080 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.155+0000 7fb8bd7fa640 1 -- 172.21.15.79:0/624576285 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8b003ab70 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.155+0000 7fb8bd7fa640 1 -- 172.21.15.79:0/624576285 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b0039cc0 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.155+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c019a520 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.155+0000 7fb8bd7fa640 1 -- 172.21.15.79:0/624576285 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7fb8b0038040 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.156+0000 7fb8bd7fa640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb8ac037860 0x7fb8ac039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.156+0000 7fb8bd7fa640 1 -- 172.21.15.79:0/624576285 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb8b006a650 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.156+0000 7fb8b7fff640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb8ac037860 0x7fb8ac039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.156+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c010a070 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.156+0000 7fb8b7fff640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb8ac037860 0x7fb8ac039d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80098c0 tx=0x7fb8a8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.160+0000 7fb8bd7fa640 1 -- 172.21.15.79:0/624576285 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb8b0041070 con 0x7fb8c0105c40 2024-06-01T00:46:20.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.334+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb8c010a230 con 0x7fb8c0105c40 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.338+0000 7fb8bd7fa640 1 -- 172.21.15.79:0/624576285 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb8b0030be0 con 0x7fb8c0105c40 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb8ac037860 msgr2=0x7fb8ac039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb8ac037860 0x7fb8ac039d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80098c0 tx=0x7fb8a8006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 msgr2=0x7fb8c0199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0199940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8b002acc0 tx=0x7fb8b0031a50 comp rx=0 tx=0).stop 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 shutdown_connections 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7fb8ac037860 0x7fb8ac039d20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 --2- 172.21.15.79:0/624576285 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8c0105c40 0x7fb8c0199940 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 >> 172.21.15.79:0/624576285 conn(0x7fb8c01011c0 msgr2=0x7fb8c0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 shutdown_connections 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:20.341+0000 7fb8c5f60640 1 -- 172.21.15.79:0/624576285 wait complete. 2024-06-01T00:46:20.918 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling the dashboard module... 2024-06-01T00:46:21.293 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:21 smithi079 ceph-mon[29356]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:21.293 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:21 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/624576285' entity='client.admin' 2024-06-01T00:46:22.473 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:22 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/964056583' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-01T00:46:22.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581aaf2640 1 Processor -- start 2024-06-01T00:46:22.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581aaf2640 1 -- start start 2024-06-01T00:46:22.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581aaf2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814104350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:22.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581aaf2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5814104920 con 0x7f5814103f30 2024-06-01T00:46:22.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581888e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814104350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:22.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581888e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814104350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39678/0 (socket says 172.21.15.79:39678) 2024-06-01T00:46:22.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581888e640 1 -- 172.21.15.79:0/3505083880 learned_addr learned my addr 172.21.15.79:0/3505083880 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.291+0000 7f581888e640 1 -- 172.21.15.79:0/3505083880 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5814106ec0 con 0x7f5814103f30 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.292+0000 7f581888e640 1 --2- 172.21.15.79:0/3505083880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814104350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5810009820 tx=0x7f581002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3080b223e250ae server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.292+0000 7f580b7fe640 1 -- 172.21.15.79:0/3505083880 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581002ad40 con 0x7f5814103f30 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.292+0000 7f580b7fe640 1 -- 172.21.15.79:0/3505083880 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5810032900 con 0x7f5814103f30 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.292+0000 7f580b7fe640 1 -- 172.21.15.79:0/3505083880 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5810030860 con 0x7f5814103f30 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 -- 172.21.15.79:0/3505083880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 msgr2=0x7f5814104350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 --2- 172.21.15.79:0/3505083880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814104350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5810009820 tx=0x7f581002a270 comp rx=0 tx=0).stop 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 -- 172.21.15.79:0/3505083880 shutdown_connections 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 --2- 172.21.15.79:0/3505083880 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814104350 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 -- 172.21.15.79:0/3505083880 >> 172.21.15.79:0/3505083880 conn(0x7f58140ff860 msgr2=0x7f5814101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 -- 172.21.15.79:0/3505083880 shutdown_connections 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 -- 172.21.15.79:0/3505083880 wait complete. 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.293+0000 7f581aaf2640 1 Processor -- start 2024-06-01T00:46:22.833 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581aaf2640 1 -- start start 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581aaf2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581aaf2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5814199da0 con 0x7f5814103f30 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581888e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581888e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:39688/0 (socket says 172.21.15.79:39688) 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581888e640 1 -- 172.21.15.79:0/964056583 learned_addr learned my addr 172.21.15.79:0/964056583 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581888e640 1 -- 172.21.15.79:0/964056583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58100094d0 con 0x7f5814103f30 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581888e640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814199860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5810009940 tx=0x7f5810032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5810032c80 con 0x7f5814103f30 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5810032de0 con 0x7f5814103f30 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5814199fa0 con 0x7f5814103f30 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.294+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f581003a730 con 0x7f5814103f30 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.295+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581419a440 con 0x7f5814103f30 2024-06-01T00:46:22.834 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.295+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 44866+0+0 (secure 0 0 0) 0x7f581003c400 con 0x7f5814103f30 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.295+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581410a0f0 con 0x7f5814103f30 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.295+0000 7f5809ffb640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f57fc037860 0x7f57fc039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.296+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f581006a950 con 0x7f5814103f30 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.296+0000 7f580bfff640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f57fc037860 0x7f57fc039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.296+0000 7f580bfff640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f57fc037860 0x7f57fc039d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58040098c0 tx=0x7f5804006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.300+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5810043310 con 0x7f5814103f30 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:21.518+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f581410a330 con 0x7f5814103f30 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.169+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 9) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f581003ad70 con 0x7f5814103f30 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.170+0000 7f5809ffb640 1 -- 172.21.15.79:0/964056583 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f581003cc50 con 0x7f5814103f30 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f57fc037860 msgr2=0x7f57fc039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f57fc037860 0x7f57fc039d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58040098c0 tx=0x7f5804006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 msgr2=0x7f5814199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:22.835 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814199860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5810009940 tx=0x7f5810032670 comp rx=0 tx=0).stop 2024-06-01T00:46:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 shutdown_connections 2024-06-01T00:46:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f57fc037860 0x7f57fc039d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 --2- 172.21.15.79:0/964056583 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5814103f30 0x7f5814199860 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 >> 172.21.15.79:0/964056583 conn(0x7f58140ff860 msgr2=0x7f58141001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 shutdown_connections 2024-06-01T00:46:22.836 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:22.173+0000 7f581aaf2640 1 -- 172.21.15.79:0/964056583 wait complete. 2024-06-01T00:46:23.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:23 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/964056583' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-01T00:46:23.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:23 smithi079 ceph-mon[29356]: mgrmap e9: smithi079.emylav(active, since 26s) 2024-06-01T00:46:24.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "smithi079.emylav", 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.254+0000 7f5bf9a62640 1 Processor -- start 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.254+0000 7f5bf9a62640 1 -- start start 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.254+0000 7f5bf9a62640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf4110a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.254+0000 7f5bf9a62640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf4111040 con 0x7f5bf4110650 2024-06-01T00:46:24.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.254+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf4110a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.254+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf4110a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36650/0 (socket says 172.21.15.79:36650) 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.254+0000 7f5bf37fe640 1 -- 172.21.15.79:0/604065921 learned_addr learned my addr 172.21.15.79:0/604065921 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.255+0000 7f5bf37fe640 1 -- 172.21.15.79:0/604065921 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf4111850 con 0x7f5bf4110650 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.255+0000 7f5bf37fe640 1 --2- 172.21.15.79:0/604065921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf4110a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009820 tx=0x7f5be402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c0911e3ff2054 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.255+0000 7f5bf27fc640 1 -- 172.21.15.79:0/604065921 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5be402ad40 con 0x7f5bf4110650 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.255+0000 7f5bf27fc640 1 -- 172.21.15.79:0/604065921 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5be4032900 con 0x7f5bf4110650 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.255+0000 7f5bf27fc640 1 -- 172.21.15.79:0/604065921 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5be4030860 con 0x7f5bf4110650 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.256+0000 7f5bf9a62640 1 -- 172.21.15.79:0/604065921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 msgr2=0x7f5bf4110a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.256+0000 7f5bf9a62640 1 --2- 172.21.15.79:0/604065921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf4110a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009820 tx=0x7f5be402a270 comp rx=0 tx=0).stop 2024-06-01T00:46:24.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.256+0000 7f5bf9a62640 1 -- 172.21.15.79:0/604065921 shutdown_connections 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.256+0000 7f5bf9a62640 1 --2- 172.21.15.79:0/604065921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf4110a70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.256+0000 7f5bf9a62640 1 -- 172.21.15.79:0/604065921 >> 172.21.15.79:0/604065921 conn(0x7f5bf406b0a0 msgr2=0x7f5bf406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.256+0000 7f5bf9a62640 1 -- 172.21.15.79:0/604065921 shutdown_connections 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.256+0000 7f5bf9a62640 1 -- 172.21.15.79:0/604065921 wait complete. 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.257+0000 7f5bf9a62640 1 Processor -- start 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.257+0000 7f5bf9a62640 1 -- start start 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.257+0000 7f5bf9a62640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf41a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.257+0000 7f5bf9a62640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf41a47f0 con 0x7f5bf4110650 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.257+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf41a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.257+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf41a42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36662/0 (socket says 172.21.15.79:36662) 2024-06-01T00:46:24.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.257+0000 7f5bf37fe640 1 -- 172.21.15.79:0/2236336432 learned_addr learned my addr 172.21.15.79:0/2236336432 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.258+0000 7f5bf37fe640 1 -- 172.21.15.79:0/2236336432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be40094d0 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.258+0000 7f5bf37fe640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf41a42b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009940 tx=0x7f5be4032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.258+0000 7f5bf0ff9640 1 -- 172.21.15.79:0/2236336432 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5be4032c80 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.258+0000 7f5bf0ff9640 1 -- 172.21.15.79:0/2236336432 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5be4032de0 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.258+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf41a49f0 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.258+0000 7f5bf0ff9640 1 -- 172.21.15.79:0/2236336432 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5be403a730 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.258+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf41a4e90 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.259+0000 7f5bf0ff9640 1 -- 172.21.15.79:0/2236336432 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f5be403a9a0 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.259+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf4114a80 con 0x7f5bf4110650 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.259+0000 7f5bf0ff9640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 0x7f5be0039dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:24.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.259+0000 7f5bf0ff9640 1 -- 172.21.15.79:0/2236336432 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5be406a930 con 0x7f5bf4110650 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.259+0000 7f5bf2ffd640 1 -- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 msgr2=0x7f5be0039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1904937508 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.259+0000 7f5bf2ffd640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 0x7f5be0039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.264+0000 7f5bf0ff9640 1 -- 172.21.15.79:0/2236336432 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5be4043310 con 0x7f5bf4110650 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.460+0000 7f5bf2ffd640 1 -- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 msgr2=0x7f5be0039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1904937508 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.460+0000 7f5bf2ffd640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 0x7f5be0039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.491+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5bf41a7b20 con 0x7f5bf4110650 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.492+0000 7f5bf0ff9640 1 -- 172.21.15.79:0/2236336432 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f5be4037ed0 con 0x7f5bf4110650 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 msgr2=0x7f5be0039dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 0x7f5be0039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 msgr2=0x7f5bf41a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:24.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf41a42b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5be4009940 tx=0x7f5be4032670 comp rx=0 tx=0).stop 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 shutdown_connections 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f5be0037900 0x7f5be0039dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 --2- 172.21.15.79:0/2236336432 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5bf4110650 0x7f5bf41a42b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 >> 172.21.15.79:0/2236336432 conn(0x7f5bf406b0a0 msgr2=0x7f5bf4112a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 shutdown_connections 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:23.494+0000 7f5bf9a62640 1 -- 172.21.15.79:0/2236336432 wait complete. 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-06-01T00:46:24.088 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 9... 2024-06-01T00:46:24.328 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:24 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2236336432' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-01T00:46:27.251 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: Active manager daemon smithi079.emylav restarted 2024-06-01T00:46:27.251 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: Activating manager daemon smithi079.emylav 2024-06-01T00:46:27.251 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: osdmap e3: 0 total, 0 up, 0 in 2024-06-01T00:46:27.251 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: mgrmap e10: smithi079.emylav(active, starting, since 0.00535671s) 2024-06-01T00:46:27.251 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr metadata", "who": "smithi079.emylav", "id": "smithi079.emylav"}]: dispatch 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: Manager daemon smithi079.emylav is now available 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/mirror_snapshot_schedule"}]: dispatch 2024-06-01T00:46:27.252 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:27 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/trash_purge_schedule"}]: dispatch 2024-06-01T00:46:28.916 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.527+0000 7f2e78132640 1 Processor -- start 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.527+0000 7f2e78132640 1 -- start start 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.527+0000 7f2e78132640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70104fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.527+0000 7f2e78132640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e70105570 con 0x7f2e70104b80 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.527+0000 7f2e75ece640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70104fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.527+0000 7f2e75ece640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70104fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36664/0 (socket says 172.21.15.79:36664) 2024-06-01T00:46:28.917 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.527+0000 7f2e75ece640 1 -- 172.21.15.79:0/1907002611 learned_addr learned my addr 172.21.15.79:0/1907002611 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e75ece640 1 -- 172.21.15.79:0/1907002611 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e70105d80 con 0x7f2e70104b80 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e75ece640 1 --2- 172.21.15.79:0/1907002611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70104fa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009820 tx=0x7f2e6c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee33392591107bcf server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e74ecc640 1 -- 172.21.15.79:0/1907002611 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6c02ad40 con 0x7f2e70104b80 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e74ecc640 1 -- 172.21.15.79:0/1907002611 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e6c032900 con 0x7f2e70104b80 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e74ecc640 1 -- 172.21.15.79:0/1907002611 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6c030860 con 0x7f2e70104b80 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e78132640 1 -- 172.21.15.79:0/1907002611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 msgr2=0x7f2e70104fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e78132640 1 --2- 172.21.15.79:0/1907002611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70104fa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009820 tx=0x7f2e6c02a270 comp rx=0 tx=0).stop 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e78132640 1 -- 172.21.15.79:0/1907002611 shutdown_connections 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e78132640 1 --2- 172.21.15.79:0/1907002611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70104fa0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e78132640 1 -- 172.21.15.79:0/1907002611 >> 172.21.15.79:0/1907002611 conn(0x7f2e701000e0 msgr2=0x7f2e70102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e78132640 1 -- 172.21.15.79:0/1907002611 shutdown_connections 2024-06-01T00:46:28.918 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.529+0000 7f2e78132640 1 -- 172.21.15.79:0/1907002611 wait complete. 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e78132640 1 Processor -- start 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e78132640 1 -- start start 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e78132640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e78132640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e70079ed0 con 0x7f2e70104b80 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e75ece640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e75ece640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70079990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36680/0 (socket says 172.21.15.79:36680) 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e75ece640 1 -- 172.21.15.79:0/2880897133 learned_addr learned my addr 172.21.15.79:0/2880897133 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.530+0000 7f2e75ece640 1 -- 172.21.15.79:0/2880897133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c0094d0 con 0x7f2e70104b80 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.531+0000 7f2e75ece640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70079990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009940 tx=0x7f2e6c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.531+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6c032c80 con 0x7f2e70104b80 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.531+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e6c032de0 con 0x7f2e70104b80 2024-06-01T00:46:28.919 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.531+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7019c610 con 0x7f2e70104b80 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.531+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7019ca50 con 0x7f2e70104b80 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.531+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e6c03a730 con 0x7f2e70104b80 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.533+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f2e6c03a9a0 con 0x7f2e70104b80 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.533+0000 7f2e66ffd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 0x7f2e5c039dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.533+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 --> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e70108fb0 con 0x7f2e5c037900 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.533+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2e6c06b8e0 con 0x7f2e70104b80 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.533+0000 7f2e756cd640 1 -- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 msgr2=0x7f2e5c039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1904937508 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.533+0000 7f2e756cd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 0x7f2e5c039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.734+0000 7f2e756cd640 1 -- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 msgr2=0x7f2e5c039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1904937508 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:24.734+0000 7f2e756cd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 0x7f2e5c039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:25.135+0000 7f2e756cd640 1 -- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 msgr2=0x7f2e5c039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1904937508 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:25.135+0000 7f2e756cd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 0x7f2e5c039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-01T00:46:28.920 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:25.936+0000 7f2e756cd640 1 -- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 msgr2=0x7f2e5c039dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/1904937508 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:25.936+0000 7f2e756cd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 0x7f2e5c039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:27.104+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 10) v1 ==== 44646+0+0 (secure 0 0 0) 0x7f2e6c06a710 con 0x7f2e70104b80 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:27.104+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 msgr2=0x7f2e5c039dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:27.104+0000 7f2e66ffd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 0x7f2e5c039dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.103+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 11) v1 ==== 44773+0+0 (secure 0 0 0) 0x7f2e6c06a9c0 con 0x7f2e70104b80 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.103+0000 7f2e66ffd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2e5c03aff0 0x7f2e5c03d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.103+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e70108fb0 con 0x7f2e5c03aff0 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.103+0000 7f2e756cd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2e5c03aff0 0x7f2e5c03d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.104+0000 7f2e756cd640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2e5c03aff0 0x7f2e5c03d3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e60003980 tx=0x7f2e600092b0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.105+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mgr.14162 v2:172.21.15.79:6800/3983389754 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f2e70108fb0 con 0x7f2e5c03aff0 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.110+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2e70109120 con 0x7f2e5c03aff0 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e66ffd640 1 -- 172.21.15.79:0/2880897133 <== mgr.14162 v2:172.21.15.79:6800/3983389754 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f2e70109120 con 0x7f2e5c03aff0 2024-06-01T00:46:28.921 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2e5c03aff0 msgr2=0x7f2e5c03d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2e5c03aff0 0x7f2e5c03d3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e60003980 tx=0x7f2e600092b0 comp rx=0 tx=0).stop 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 msgr2=0x7f2e70079990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70079990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009940 tx=0x7f2e6c032670 comp rx=0 tx=0).stop 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 shutdown_connections 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2e5c03aff0 0x7f2e5c03d3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:6800/1904937508,v1:172.21.15.79:6801/1904937508] conn(0x7f2e5c037900 0x7f2e5c039dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 --2- 172.21.15.79:0/2880897133 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2e70104b80 0x7f2e70079990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:28.922 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 >> 172.21.15.79:0/2880897133 conn(0x7f2e701000e0 msgr2=0x7f2e70100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:28.923 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 shutdown_connections 2024-06-01T00:46:28.923 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:28.111+0000 7f2e78132640 1 -- 172.21.15.79:0/2880897133 wait complete. 2024-06-01T00:46:28.923 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 9 is available 2024-06-01T00:46:28.923 INFO:teuthology.orchestra.run.smithi079.stdout:Generating a dashboard self-signed certificate... 2024-06-01T00:46:29.158 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:29 smithi079 ceph-mon[29356]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-01T00:46:29.159 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:29 smithi079 ceph-mon[29356]: mgrmap e11: smithi079.emylav(active, since 1.00545s) 2024-06-01T00:46:29.159 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:29 smithi079 ceph-mon[29356]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-01T00:46:29.159 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:29 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:29.159 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:29 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb578b6b640 1 Processor -- start 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb578b6b640 1 -- start start 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb578b6b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574105440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb578b6b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574105a10 con 0x7fb574105020 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb57259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574105440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb57259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574105440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36752/0 (socket says 172.21.15.79:36752) 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb57259c640 1 -- 172.21.15.79:0/554158799 learned_addr learned my addr 172.21.15.79:0/554158799 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.355+0000 7fb57259c640 1 -- 172.21.15.79:0/554158799 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb574105d40 con 0x7fb574105020 2024-06-01T00:46:30.274 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.356+0000 7fb57259c640 1 --2- 172.21.15.79:0/554158799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574105440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb564009820 tx=0x7fb56402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f865141588249f server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.356+0000 7fb57159a640 1 -- 172.21.15.79:0/554158799 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb56402ad40 con 0x7fb574105020 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.356+0000 7fb57159a640 1 -- 172.21.15.79:0/554158799 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb564032900 con 0x7fb574105020 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.356+0000 7fb57159a640 1 -- 172.21.15.79:0/554158799 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb564030860 con 0x7fb574105020 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.357+0000 7fb578b6b640 1 -- 172.21.15.79:0/554158799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 msgr2=0x7fb574105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.357+0000 7fb578b6b640 1 --2- 172.21.15.79:0/554158799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574105440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb564009820 tx=0x7fb56402a270 comp rx=0 tx=0).stop 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.357+0000 7fb578b6b640 1 -- 172.21.15.79:0/554158799 shutdown_connections 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.357+0000 7fb578b6b640 1 --2- 172.21.15.79:0/554158799 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574105440 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.357+0000 7fb578b6b640 1 -- 172.21.15.79:0/554158799 >> 172.21.15.79:0/554158799 conn(0x7fb5740ff820 msgr2=0x7fb574101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.357+0000 7fb578b6b640 1 -- 172.21.15.79:0/554158799 shutdown_connections 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.357+0000 7fb578b6b640 1 -- 172.21.15.79:0/554158799 wait complete. 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb578b6b640 1 Processor -- start 2024-06-01T00:46:30.275 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb578b6b640 1 -- start start 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb578b6b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb578b6b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574198c70 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb57259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb57259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36754/0 (socket says 172.21.15.79:36754) 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb57259c640 1 -- 172.21.15.79:0/1237746018 learned_addr learned my addr 172.21.15.79:0/1237746018 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.358+0000 7fb57259c640 1 -- 172.21.15.79:0/1237746018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5640094d0 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.359+0000 7fb57259c640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574198730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb564032670 tx=0x7fb5640326a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.359+0000 7fb5637fe640 1 -- 172.21.15.79:0/1237746018 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb564032c80 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.359+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb574198e70 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.359+0000 7fb5637fe640 1 -- 172.21.15.79:0/1237746018 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb564032de0 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.359+0000 7fb5637fe640 1 -- 172.21.15.79:0/1237746018 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb56403a800 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.359+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb574199290 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.360+0000 7fb5637fe640 1 -- 172.21.15.79:0/1237746018 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 11) v1 ==== 44773+0+0 (secure 0 0 0) 0x7fb56403aa70 con 0x7fb574105020 2024-06-01T00:46:30.276 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.360+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb574108f70 con 0x7fb574105020 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.360+0000 7fb5637fe640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb55c0399a0 0x7fb55c03be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.360+0000 7fb5637fe640 1 -- 172.21.15.79:0/1237746018 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb56406aa60 con 0x7fb574105020 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.360+0000 7fb571d9b640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb55c0399a0 0x7fb55c03be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.360+0000 7fb571d9b640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb55c0399a0 0x7fb55c03be60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5680098c0 tx=0x7fb568006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.365+0000 7fb5637fe640 1 -- 172.21.15.79:0/1237746018 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb564039c60 con 0x7fb574105020 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.561+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fb574100910 con 0x7fb55c0399a0 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.655+0000 7fb5637fe640 1 -- 172.21.15.79:0/1237746018 <== mgr.14162 v2:172.21.15.79:6800/3983389754 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fb574100910 con 0x7fb55c0399a0 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb55c0399a0 msgr2=0x7fb55c03be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:30.277 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb55c0399a0 0x7fb55c03be60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5680098c0 tx=0x7fb568006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 msgr2=0x7fb574198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574198730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb564032670 tx=0x7fb5640326a0 comp rx=0 tx=0).stop 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 shutdown_connections 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb55c0399a0 0x7fb55c03be60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 --2- 172.21.15.79:0/1237746018 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb574105020 0x7fb574198730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 >> 172.21.15.79:0/1237746018 conn(0x7fb5740ff820 msgr2=0x7fb574100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 shutdown_connections 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:29.657+0000 7fb578b6b640 1 -- 172.21.15.79:0/1237746018 wait complete. 2024-06-01T00:46:30.278 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial admin user... 2024-06-01T00:46:30.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: [01/Jun/2024:00:46:28] ENGINE Bus STARTING 2024-06-01T00:46:30.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: [01/Jun/2024:00:46:28] ENGINE Serving on http://172.21.15.79:8765 2024-06-01T00:46:30.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: [01/Jun/2024:00:46:28] ENGINE Serving on https://172.21.15.79:7150 2024-06-01T00:46:30.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: [01/Jun/2024:00:46:28] ENGINE Bus STARTED 2024-06-01T00:46:30.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: [01/Jun/2024:00:46:28] ENGINE Error in HTTPServer.serve 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Traceback (most recent call last): 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._run(expiration_interval) 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: s = self.context.wrap_socket( 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-01T00:46:30.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: return self.sslsocket_class._create( 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self.do_handshake() 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._sslobj.do_handshake() 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:30.358 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:30 smithi079 ceph-mon[29356]: mgrmap e12: smithi079.emylav(active, since 2s) 2024-06-01T00:46:31.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:31 smithi079 ceph-mon[29356]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:31.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:31 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:31.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$PYp81CfZsM/xZcHf.JH3YeCbWpbjO/FUbMPR7LKNL79ie7Uzcn23O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717202791, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-01T00:46:31.676 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.666+0000 7f63cd2d3640 1 Processor -- start 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.666+0000 7f63cd2d3640 1 -- start start 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.666+0000 7f63cd2d3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.666+0000 7f63cd2d3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8079e40 con 0x7f63c8106680 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.667+0000 7f63c6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.667+0000 7f63c6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36756/0 (socket says 172.21.15.79:36756) 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.667+0000 7f63c6ffd640 1 -- 172.21.15.79:0/3118109553 learned_addr learned my addr 172.21.15.79:0/3118109553 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.667+0000 7f63c6ffd640 1 -- 172.21.15.79:0/3118109553 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c8108fb0 con 0x7f63c8106680 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63c6ffd640 1 --2- 172.21.15.79:0/3118109553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8108a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009820 tx=0x7f63b802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf931b3a47a4910 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63c5ffb640 1 -- 172.21.15.79:0/3118109553 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b802ad40 con 0x7f63c8106680 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63c5ffb640 1 -- 172.21.15.79:0/3118109553 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63b8032900 con 0x7f63c8106680 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63c5ffb640 1 -- 172.21.15.79:0/3118109553 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8030860 con 0x7f63c8106680 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63cd2d3640 1 -- 172.21.15.79:0/3118109553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 msgr2=0x7f63c8108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:31.677 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63cd2d3640 1 --2- 172.21.15.79:0/3118109553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8108a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009820 tx=0x7f63b802a270 comp rx=0 tx=0).stop 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63cd2d3640 1 -- 172.21.15.79:0/3118109553 shutdown_connections 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63cd2d3640 1 --2- 172.21.15.79:0/3118109553 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8108a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63cd2d3640 1 -- 172.21.15.79:0/3118109553 >> 172.21.15.79:0/3118109553 conn(0x7f63c81000d0 msgr2=0x7f63c81024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63cd2d3640 1 -- 172.21.15.79:0/3118109553 shutdown_connections 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.668+0000 7f63cd2d3640 1 -- 172.21.15.79:0/3118109553 wait complete. 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.669+0000 7f63cd2d3640 1 Processor -- start 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.669+0000 7f63cd2d3640 1 -- start start 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.669+0000 7f63cd2d3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.669+0000 7f63cd2d3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c8199db0 con 0x7f63c8106680 2024-06-01T00:46:31.678 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.669+0000 7f63c6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.669+0000 7f63c6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36766/0 (socket says 172.21.15.79:36766) 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.669+0000 7f63c6ffd640 1 -- 172.21.15.79:0/2410158460 learned_addr learned my addr 172.21.15.79:0/2410158460 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.670+0000 7f63c6ffd640 1 -- 172.21.15.79:0/2410158460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b80094d0 con 0x7f63c8106680 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.670+0000 7f63c6ffd640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8199870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f63b8032670 tx=0x7f63b80326a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.670+0000 7f63affff640 1 -- 172.21.15.79:0/2410158460 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b8032c80 con 0x7f63c8106680 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.670+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c8199fb0 con 0x7f63c8106680 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.670+0000 7f63affff640 1 -- 172.21.15.79:0/2410158460 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63b8032de0 con 0x7f63c8106680 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.670+0000 7f63affff640 1 -- 172.21.15.79:0/2410158460 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63b803a7c0 con 0x7f63c8106680 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.670+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c819a450 con 0x7f63c8106680 2024-06-01T00:46:31.679 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.671+0000 7f63affff640 1 -- 172.21.15.79:0/2410158460 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) v1 ==== 44879+0+0 (secure 0 0 0) 0x7f63b803aa00 con 0x7f63c8106680 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.671+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c8109e10 con 0x7f63c8106680 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.671+0000 7f63affff640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f63b00378b0 0x7f63b0039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.671+0000 7f63affff640 1 -- 172.21.15.79:0/2410158460 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f63b806a9f0 con 0x7f63c8106680 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.671+0000 7f63c67fc640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f63b00378b0 0x7f63b0039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.671+0000 7f63c67fc640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f63b00378b0 0x7f63b0039d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0098c0 tx=0x7f63bc006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.676+0000 7f63affff640 1 -- 172.21.15.79:0/2410158460 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f63b8039ca0 con 0x7f63c8106680 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:30.874+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f63c8061900 con 0x7f63b00378b0 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.098+0000 7f63affff640 1 -- 172.21.15.79:0/2410158460 <== mgr.14162 v2:172.21.15.79:6800/3983389754 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f63c8061900 con 0x7f63b00378b0 2024-06-01T00:46:31.680 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.100+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f63b00378b0 msgr2=0x7f63b0039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.100+0000 7f63cd2d3640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f63b00378b0 0x7f63b0039d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0098c0 tx=0x7f63bc006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.100+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 msgr2=0x7f63c8199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.100+0000 7f63cd2d3640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8199870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f63b8032670 tx=0x7f63b80326a0 comp rx=0 tx=0).stop 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.100+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 shutdown_connections 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.100+0000 7f63cd2d3640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f63b00378b0 0x7f63b0039d70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.100+0000 7f63cd2d3640 1 --2- 172.21.15.79:0/2410158460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63c8106680 0x7f63c8199870 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.101+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 >> 172.21.15.79:0/2410158460 conn(0x7f63c81000d0 msgr2=0x7f63c8100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.101+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 shutdown_connections 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:31.101+0000 7f63cd2d3640 1 -- 172.21.15.79:0/2410158460 wait complete. 2024-06-01T00:46:31.681 INFO:teuthology.orchestra.run.smithi079.stdout:Fetching dashboard port number... 2024-06-01T00:46:32.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:32 smithi079 ceph-mon[29356]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:33.285 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:33 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1586750193' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-01T00:46:33.292 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 8443 2024-06-01T00:46:33.292 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe39e22b640 1 Processor -- start 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe39e22b640 1 -- start start 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe39e22b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe398106060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe39e22b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe398106630 con 0x7fe398105c40 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe3977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe398106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe3977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe398106060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51350/0 (socket says 172.21.15.79:51350) 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe3977fe640 1 -- 172.21.15.79:0/367106839 learned_addr learned my addr 172.21.15.79:0/367106839 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.517+0000 7fe3977fe640 1 -- 172.21.15.79:0/367106839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe398106e40 con 0x7fe398105c40 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.518+0000 7fe3977fe640 1 --2- 172.21.15.79:0/367106839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe398106060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe388009820 tx=0x7fe38802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0358f7ad82d9f server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.518+0000 7fe396ffd640 1 -- 172.21.15.79:0/367106839 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe38802ad40 con 0x7fe398105c40 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.518+0000 7fe396ffd640 1 -- 172.21.15.79:0/367106839 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe388032900 con 0x7fe398105c40 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.518+0000 7fe396ffd640 1 -- 172.21.15.79:0/367106839 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe388030860 con 0x7fe398105c40 2024-06-01T00:46:33.293 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 -- 172.21.15.79:0/367106839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 msgr2=0x7fe398106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 --2- 172.21.15.79:0/367106839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe398106060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe388009820 tx=0x7fe38802a270 comp rx=0 tx=0).stop 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 -- 172.21.15.79:0/367106839 shutdown_connections 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 --2- 172.21.15.79:0/367106839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe398106060 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 -- 172.21.15.79:0/367106839 >> 172.21.15.79:0/367106839 conn(0x7fe3981011c0 msgr2=0x7fe398103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 -- 172.21.15.79:0/367106839 shutdown_connections 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 -- 172.21.15.79:0/367106839 wait complete. 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 Processor -- start 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 -- start start 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe3981998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.519+0000 7fe39e22b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe398199de0 con 0x7fe398105c40 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe3981998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:33.294 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3977fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe3981998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51356/0 (socket says 172.21.15.79:51356) 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3977fe640 1 -- 172.21.15.79:0/1586750193 learned_addr learned my addr 172.21.15.79:0/1586750193 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3977fe640 1 -- 172.21.15.79:0/1586750193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3880094d0 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3977fe640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe3981998a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe3880097f0 tx=0x7fe388030e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3957fa640 1 -- 172.21.15.79:0/1586750193 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe388040070 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe398199fe0 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3957fa640 1 -- 172.21.15.79:0/1586750193 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe388032dd0 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe3957fa640 1 -- 172.21.15.79:0/1586750193 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe38803a800 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.520+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39819a480 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.521+0000 7fe3957fa640 1 -- 172.21.15.79:0/1586750193 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) v1 ==== 44879+0+0 (secure 0 0 0) 0x7fe38803a960 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.521+0000 7fe3957fa640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fe3840378b0 0x7fe384039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.521+0000 7fe3957fa640 1 -- 172.21.15.79:0/1586750193 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe38806a700 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.521+0000 7fe38ffff640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fe3840378b0 0x7fe384039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.521+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe398061ae0 con 0x7fe398105c40 2024-06-01T00:46:33.295 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.521+0000 7fe38ffff640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fe3840378b0 0x7fe384039d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe3800098c0 tx=0x7fe380006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.526+0000 7fe3957fa640 1 -- 172.21.15.79:0/1586750193 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe388039b40 con 0x7fe398105c40 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.696+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe39810a070 con 0x7fe398105c40 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.696+0000 7fe3957fa640 1 -- 172.21.15.79:0/1586750193 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe388031e70 con 0x7fe398105c40 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fe3840378b0 msgr2=0x7fe384039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fe3840378b0 0x7fe384039d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe3800098c0 tx=0x7fe380006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 msgr2=0x7fe3981998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe3981998a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe3880097f0 tx=0x7fe388030e90 comp rx=0 tx=0).stop 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 shutdown_connections 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fe3840378b0 0x7fe384039d70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 --2- 172.21.15.79:0/1586750193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe398105c40 0x7fe3981998a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.698+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 >> 172.21.15.79:0/1586750193 conn(0x7fe3981011c0 msgr2=0x7fe398101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.699+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 shutdown_connections 2024-06-01T00:46:33.296 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:32.699+0000 7fe39e22b640 1 -- 172.21.15.79:0/1586750193 wait complete. 2024-06-01T00:46:33.299 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-01T00:46:33.299 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-06-01T00:46:33.306 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-01T00:46:33.307 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-06-01T00:46:33.307 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-06-01T00:46:33.307 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph Dashboard is now available at: 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout: URL: https://smithi079.front.sepia.ceph.com:8443/ 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout: User: admin 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout: Password: f3eg8lseqp 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout:Saving cluster configuration to /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config directory 2024-06-01T00:46:33.309 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling autotune for osd_memory_target 2024-06-01T00:46:34.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:34 smithi079 ceph-mon[29356]: mgrmap e13: smithi079.emylav(active, since 6s) 2024-06-01T00:46:34.486 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c31894640 1 Processor -- start 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c31894640 1 -- start start 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c31894640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c108ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c31894640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c079de0 con 0x7f5c2c1066e0 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c108ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c108ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51362/0 (socket says 172.21.15.79:51362) 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c2affd640 1 -- 172.21.15.79:0/4244632212 learned_addr learned my addr 172.21.15.79:0/4244632212 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.717+0000 7f5c2affd640 1 -- 172.21.15.79:0/4244632212 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c109010 con 0x7f5c2c1066e0 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.718+0000 7f5c2affd640 1 --2- 172.21.15.79:0/4244632212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c108ad0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009820 tx=0x7f5c1c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9c240d3f95b574 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:34.487 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.718+0000 7f5c2a7fc640 1 -- 172.21.15.79:0/4244632212 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c1c02ad40 con 0x7f5c2c1066e0 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.718+0000 7f5c2a7fc640 1 -- 172.21.15.79:0/4244632212 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c1c032900 con 0x7f5c2c1066e0 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.718+0000 7f5c2a7fc640 1 -- 172.21.15.79:0/4244632212 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c1c030860 con 0x7f5c2c1066e0 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.719+0000 7f5c31894640 1 -- 172.21.15.79:0/4244632212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 msgr2=0x7f5c2c108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.719+0000 7f5c31894640 1 --2- 172.21.15.79:0/4244632212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c108ad0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009820 tx=0x7f5c1c02a270 comp rx=0 tx=0).stop 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.719+0000 7f5c31894640 1 -- 172.21.15.79:0/4244632212 shutdown_connections 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.719+0000 7f5c31894640 1 --2- 172.21.15.79:0/4244632212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c108ad0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.719+0000 7f5c31894640 1 -- 172.21.15.79:0/4244632212 >> 172.21.15.79:0/4244632212 conn(0x7f5c2c1000d0 msgr2=0x7f5c2c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.719+0000 7f5c31894640 1 -- 172.21.15.79:0/4244632212 shutdown_connections 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.719+0000 7f5c31894640 1 -- 172.21.15.79:0/4244632212 wait complete. 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c31894640 1 Processor -- start 2024-06-01T00:46:34.488 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c31894640 1 -- start start 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c31894640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c31894640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c199dc0 con 0x7f5c2c1066e0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c2affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51364/0 (socket says 172.21.15.79:51364) 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c2affd640 1 -- 172.21.15.79:0/268983704 learned_addr learned my addr 172.21.15.79:0/268983704 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c2affd640 1 -- 172.21.15.79:0/268983704 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0094d0 con 0x7f5c2c1066e0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c2affd640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c199880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c032670 tx=0x7f5c1c0326a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c30892640 1 -- 172.21.15.79:0/268983704 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c1c032c80 con 0x7f5c2c1066e0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c30892640 1 -- 172.21.15.79:0/268983704 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c1c032de0 con 0x7f5c2c1066e0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c30892640 1 -- 172.21.15.79:0/268983704 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c1c03a7c0 con 0x7f5c2c1066e0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.720+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c199fc0 con 0x7f5c2c1066e0 2024-06-01T00:46:34.489 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.721+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c19a460 con 0x7f5c2c1066e0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.721+0000 7f5c30892640 1 -- 172.21.15.79:0/268983704 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f5c1c03aa00 con 0x7f5c2c1066e0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.721+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2c109e10 con 0x7f5c2c1066e0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.722+0000 7f5c30892640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5c18037950 0x7f5c18039e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.722+0000 7f5c30892640 1 -- 172.21.15.79:0/268983704 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5c1c06abf0 con 0x7f5c2c1066e0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.722+0000 7f5c23fff640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5c18037950 0x7f5c18039e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.722+0000 7f5c23fff640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5c18037950 0x7f5c18039e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5c140098c0 tx=0x7f5c14006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.726+0000 7f5c30892640 1 -- 172.21.15.79:0/268983704 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5c1c031d30 con 0x7f5c2c1066e0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.900+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f5c2c10a050 con 0x7f5c2c1066e0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.900+0000 7f5c30892640 1 -- 172.21.15.79:0/268983704 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5c1c031b50 con 0x7f5c2c1066e0 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5c18037950 msgr2=0x7f5c18039e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:34.490 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5c18037950 0x7f5c18039e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5c140098c0 tx=0x7f5c14006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 msgr2=0x7f5c2c199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c199880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c032670 tx=0x7f5c1c0326a0 comp rx=0 tx=0).stop 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 shutdown_connections 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5c18037950 0x7f5c18039e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 --2- 172.21.15.79:0/268983704 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c2c1066e0 0x7f5c2c199880 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.902+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 >> 172.21.15.79:0/268983704 conn(0x7f5c2c1000d0 msgr2=0x7f5c2c100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.903+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 shutdown_connections 2024-06-01T00:46:34.491 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:33.903+0000 7f5c31894640 1 -- 172.21.15.79:0/268983704 wait complete. 2024-06-01T00:46:35.738 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.854+0000 7f7552916640 1 Processor -- start 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.854+0000 7f7552916640 1 -- start start 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.854+0000 7f7552916640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c108b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.854+0000 7f7552916640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c079de0 con 0x7f754c106730 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.855+0000 7f754bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c108b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.855+0000 7f754bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c108b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51368/0 (socket says 172.21.15.79:51368) 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.855+0000 7f754bfff640 1 -- 172.21.15.79:0/1741110695 learned_addr learned my addr 172.21.15.79:0/1741110695 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.855+0000 7f754bfff640 1 -- 172.21.15.79:0/1741110695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c109060 con 0x7f754c106730 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f754bfff640 1 --2- 172.21.15.79:0/1741110695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c108b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f753c009820 tx=0x7f753c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24feaf03852e0ac2 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f754b7fe640 1 -- 172.21.15.79:0/1741110695 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f753c02ad40 con 0x7f754c106730 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f754b7fe640 1 -- 172.21.15.79:0/1741110695 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f753c032900 con 0x7f754c106730 2024-06-01T00:46:35.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f754b7fe640 1 -- 172.21.15.79:0/1741110695 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f753c030860 con 0x7f754c106730 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f7552916640 1 -- 172.21.15.79:0/1741110695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 msgr2=0x7f754c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f7552916640 1 --2- 172.21.15.79:0/1741110695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c108b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f753c009820 tx=0x7f753c02a270 comp rx=0 tx=0).stop 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f7552916640 1 -- 172.21.15.79:0/1741110695 shutdown_connections 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f7552916640 1 --2- 172.21.15.79:0/1741110695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c108b20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f7552916640 1 -- 172.21.15.79:0/1741110695 >> 172.21.15.79:0/1741110695 conn(0x7f754c1000e0 msgr2=0x7f754c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f7552916640 1 -- 172.21.15.79:0/1741110695 shutdown_connections 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.856+0000 7f7552916640 1 -- 172.21.15.79:0/1741110695 wait complete. 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f7552916640 1 Processor -- start 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f7552916640 1 -- start start 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f7552916640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f7552916640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c199df0 con 0x7f754c106730 2024-06-01T00:46:35.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f754bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f754bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c1998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51376/0 (socket says 172.21.15.79:51376) 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f754bfff640 1 -- 172.21.15.79:0/2455631111 learned_addr learned my addr 172.21.15.79:0/2455631111 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.857+0000 7f754bfff640 1 -- 172.21.15.79:0/2455631111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0094d0 con 0x7f754c106730 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.858+0000 7f754bfff640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c1998b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f753c032670 tx=0x7f753c0326a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.858+0000 7f7549ffb640 1 -- 172.21.15.79:0/2455631111 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f753c032c80 con 0x7f754c106730 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.858+0000 7f7549ffb640 1 -- 172.21.15.79:0/2455631111 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f753c032de0 con 0x7f754c106730 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.858+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c199ff0 con 0x7f754c106730 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.858+0000 7f7549ffb640 1 -- 172.21.15.79:0/2455631111 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f753c03a7c0 con 0x7f754c106730 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.858+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c19a490 con 0x7f754c106730 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.859+0000 7f7549ffb640 1 -- 172.21.15.79:0/2455631111 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f753c03a9b0 con 0x7f754c106730 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.859+0000 7f7549ffb640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f75380379a0 0x7f7538039e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:35.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.859+0000 7f7549ffb640 1 -- 172.21.15.79:0/2455631111 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f753c06aac0 con 0x7f754c106730 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.859+0000 7f7543fff640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f75380379a0 0x7f7538039e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.859+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c109e60 con 0x7f754c106730 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.859+0000 7f7543fff640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f75380379a0 0x7f7538039e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f75340098c0 tx=0x7f7534006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:34.864+0000 7f7549ffb640 1 -- 172.21.15.79:0/2455631111 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f753c043310 con 0x7f754c106730 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.122+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f754c109ff0 con 0x7f754c106730 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.124+0000 7f7549ffb640 1 -- 172.21.15.79:0/2455631111 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f753c039ed0 con 0x7f754c106730 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f75380379a0 msgr2=0x7f7538039e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:35.742 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f75380379a0 0x7f7538039e60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f75340098c0 tx=0x7f7534006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 msgr2=0x7f754c1998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c1998b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f753c032670 tx=0x7f753c0326a0 comp rx=0 tx=0).stop 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 shutdown_connections 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f75380379a0 0x7f7538039e60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 --2- 172.21.15.79:0/2455631111 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f754c106730 0x7f754c1998b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 >> 172.21.15.79:0/2455631111 conn(0x7f754c1000e0 msgr2=0x7f754c100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.126+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 shutdown_connections 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-06-01T00:46:35.127+0000 7f7552916640 1 -- 172.21.15.79:0/2455631111 wait complete. 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout:Or, if you are only running a single cluster on this host: 2024-06-01T00:46:35.743 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: ceph telemetry on 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout:For more information see: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:35.744 INFO:teuthology.orchestra.run.smithi079.stdout:Bootstrap complete. 2024-06-01T00:46:35.767 INFO:tasks.cephadm:Fetching config... 2024-06-01T00:46:35.768 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:46:35.768 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-01T00:46:35.824 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-01T00:46:35.824 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:46:35.824 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-01T00:46:35.879 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-01T00:46:35.880 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:46:35.880 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/keyring of=/dev/stdout 2024-06-01T00:46:35.945 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-01T00:46:35.945 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:46:35.945 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-01T00:46:36.001 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-01T00:46:36.001 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR44OWYFqe2zC5QDBHiEBFZwZ0PrBG3RCk1GPVU/sn/Sbbq4wnAt+LODM/Wr94Wxs9C/5dLJ3LPa+MiYySxqqmTpEzEuT8KxghK/37azfQi3ETtCFRrOxks0BS/070PXUa7eRcxbXucE1qJJXB2xbfmGbjZC43pI0D4g7n0bb7K1/SQNzOeAWTAhJLDrxUFi6wJWj+0Xbi8h10EaatoXC9I9KX9iKv69hAHlbv8uEkd9Tv7MjZzQhAK5+VTG97lAyjUv4+LO5jp8MRAECgV/RXr1aU8YNWUoi6Ohrv1Zeo7DOO9QuFF2MouoCj+6QV9IDHytnQ0t0h4+EKng9HJb6MuSm00u/t49UekfDCXF/bLa0IqMzRRslOGbtgcXy2rm9wXlLqloLR3YjHirxRBQ8G6Vm9xmHS1aTK5HBH+b64IVHBYpE2DVIaS5v2BM3D4+CUMKgtBoX8fw59wNEPqZukm16Jv3FuIsFsPrK8FCD/eaeuwNrP3Bwo2yeh6fk6uYM= ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-01T00:46:36.077 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR44OWYFqe2zC5QDBHiEBFZwZ0PrBG3RCk1GPVU/sn/Sbbq4wnAt+LODM/Wr94Wxs9C/5dLJ3LPa+MiYySxqqmTpEzEuT8KxghK/37azfQi3ETtCFRrOxks0BS/070PXUa7eRcxbXucE1qJJXB2xbfmGbjZC43pI0D4g7n0bb7K1/SQNzOeAWTAhJLDrxUFi6wJWj+0Xbi8h10EaatoXC9I9KX9iKv69hAHlbv8uEkd9Tv7MjZzQhAK5+VTG97lAyjUv4+LO5jp8MRAECgV/RXr1aU8YNWUoi6Ohrv1Zeo7DOO9QuFF2MouoCj+6QV9IDHytnQ0t0h4+EKng9HJb6MuSm00u/t49UekfDCXF/bLa0IqMzRRslOGbtgcXy2rm9wXlLqloLR3YjHirxRBQ8G6Vm9xmHS1aTK5HBH+b64IVHBYpE2DVIaS5v2BM3D4+CUMKgtBoX8fw59wNEPqZukm16Jv3FuIsFsPrK8FCD/eaeuwNrP3Bwo2yeh6fk6uYM= ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:46:36.088 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR44OWYFqe2zC5QDBHiEBFZwZ0PrBG3RCk1GPVU/sn/Sbbq4wnAt+LODM/Wr94Wxs9C/5dLJ3LPa+MiYySxqqmTpEzEuT8KxghK/37azfQi3ETtCFRrOxks0BS/070PXUa7eRcxbXucE1qJJXB2xbfmGbjZC43pI0D4g7n0bb7K1/SQNzOeAWTAhJLDrxUFi6wJWj+0Xbi8h10EaatoXC9I9KX9iKv69hAHlbv8uEkd9Tv7MjZzQhAK5+VTG97lAyjUv4+LO5jp8MRAECgV/RXr1aU8YNWUoi6Ohrv1Zeo7DOO9QuFF2MouoCj+6QV9IDHytnQ0t0h4+EKng9HJb6MuSm00u/t49UekfDCXF/bLa0IqMzRRslOGbtgcXy2rm9wXlLqloLR3YjHirxRBQ8G6Vm9xmHS1aTK5HBH+b64IVHBYpE2DVIaS5v2BM3D4+CUMKgtBoX8fw59wNEPqZukm16Jv3FuIsFsPrK8FCD/eaeuwNrP3Bwo2yeh6fk6uYM= ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-01T00:46:36.130 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR44OWYFqe2zC5QDBHiEBFZwZ0PrBG3RCk1GPVU/sn/Sbbq4wnAt+LODM/Wr94Wxs9C/5dLJ3LPa+MiYySxqqmTpEzEuT8KxghK/37azfQi3ETtCFRrOxks0BS/070PXUa7eRcxbXucE1qJJXB2xbfmGbjZC43pI0D4g7n0bb7K1/SQNzOeAWTAhJLDrxUFi6wJWj+0Xbi8h10EaatoXC9I9KX9iKv69hAHlbv8uEkd9Tv7MjZzQhAK5+VTG97lAyjUv4+LO5jp8MRAECgV/RXr1aU8YNWUoi6Ohrv1Zeo7DOO9QuFF2MouoCj+6QV9IDHytnQ0t0h4+EKng9HJb6MuSm00u/t49UekfDCXF/bLa0IqMzRRslOGbtgcXy2rm9wXlLqloLR3YjHirxRBQ8G6Vm9xmHS1aTK5HBH+b64IVHBYpE2DVIaS5v2BM3D4+CUMKgtBoX8fw59wNEPqZukm16Jv3FuIsFsPrK8FCD/eaeuwNrP3Bwo2yeh6fk6uYM= ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:46:36.143 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-01T00:46:36.342 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:46:36.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:36 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2455631111' entity='client.admin' 2024-06-01T00:46:38.851 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:38.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:38.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:46:38.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:38.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-01T00:46:38.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-06-01T00:46:38.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:46:38.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:38 smithi079 ceph-mon[29356]: Deploying daemon ceph-exporter.smithi079 on smithi079 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 -- 172.21.15.79:0/1311108752 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 msgr2=0x7f06d4100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 --2- 172.21.15.79:0/1311108752 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d4100290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009880 tx=0x7f06d002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 -- 172.21.15.79:0/1311108752 shutdown_connections 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 --2- 172.21.15.79:0/1311108752 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d4100290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 -- 172.21.15.79:0/1311108752 >> 172.21.15.79:0/1311108752 conn(0x7f06d40fb460 msgr2=0x7f06d40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 -- 172.21.15.79:0/1311108752 shutdown_connections 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 -- 172.21.15.79:0/1311108752 wait complete. 2024-06-01T00:46:38.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.934+0000 7f06db686640 1 Processor -- start 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06db686640 1 -- start start 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06db686640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d41953f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06db686640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d4195930 con 0x7f06d40ffe70 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06d9422640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d41953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06d9422640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d41953f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51400/0 (socket says 172.21.15.79:51400) 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06d9422640 1 -- 172.21.15.79:0/3916973272 learned_addr learned my addr 172.21.15.79:0/3916973272 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06d9422640 1 -- 172.21.15.79:0/3916973272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d0009560 con 0x7f06d40ffe70 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06d9422640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d41953f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f06d0005610 tx=0x7f06d0005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06ca7fc640 1 -- 172.21.15.79:0/3916973272 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06d0004b20 con 0x7f06d40ffe70 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06ca7fc640 1 -- 172.21.15.79:0/3916973272 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06d0004c80 con 0x7f06d40ffe70 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.935+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d4195b30 con 0x7f06d40ffe70 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.936+0000 7f06ca7fc640 1 -- 172.21.15.79:0/3916973272 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06d0031a90 con 0x7f06d40ffe70 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.936+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d4195fd0 con 0x7f06d40ffe70 2024-06-01T00:46:38.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.937+0000 7f06ca7fc640 1 -- 172.21.15.79:0/3916973272 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f06d0031d00 con 0x7f06d40ffe70 2024-06-01T00:46:38.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.937+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d4061ae0 con 0x7f06d40ffe70 2024-06-01T00:46:38.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.937+0000 7f06ca7fc640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f06c00378e0 0x7f06c0039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:38.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.937+0000 7f06ca7fc640 1 -- 172.21.15.79:0/3916973272 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f06d006b430 con 0x7f06d40ffe70 2024-06-01T00:46:38.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.937+0000 7f06d8c21640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f06c00378e0 0x7f06c0039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:38.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.937+0000 7f06d8c21640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f06c00378e0 0x7f06c0039da0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f06c40098c0 tx=0x7f06c4006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:38.953 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:38.945+0000 7f06ca7fc640 1 -- 172.21.15.79:0/3916973272 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f06d004d530 con 0x7f06d40ffe70 2024-06-01T00:46:39.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.128+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f06d4105f10 con 0x7f06d40ffe70 2024-06-01T00:46:39.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.131+0000 7f06ca7fc640 1 -- 172.21.15.79:0/3916973272 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f06d0035070 con 0x7f06d40ffe70 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.133+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f06c00378e0 msgr2=0x7f06c0039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.133+0000 7f06db686640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f06c00378e0 0x7f06c0039da0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f06c40098c0 tx=0x7f06c4006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.133+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 msgr2=0x7f06d41953f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.134+0000 7f06db686640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d41953f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f06d0005610 tx=0x7f06d0005640 comp rx=0 tx=0).stop 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.134+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 shutdown_connections 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.134+0000 7f06db686640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f06c00378e0 0x7f06c0039da0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.134+0000 7f06db686640 1 --2- 172.21.15.79:0/3916973272 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06d40ffe70 0x7f06d41953f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.134+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 >> 172.21.15.79:0/3916973272 conn(0x7f06d40fb460 msgr2=0x7f06d40fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:39.135 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.134+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 shutdown_connections 2024-06-01T00:46:39.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:39.134+0000 7f06db686640 1 -- 172.21.15.79:0/3916973272 wait complete. 2024-06-01T00:46:39.909 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-01T00:46:39.910 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-01T00:46:40.300 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:40 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3916973272' entity='client.admin' 2024-06-01T00:46:40.973 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 -- 172.21.15.79:0/2012257475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 msgr2=0x7f14940ff1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 --2- 172.21.15.79:0/2012257475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f14940ff1f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1484009880 tx=0x7f148402a5d0 comp rx=0 tx=0).stop 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 -- 172.21.15.79:0/2012257475 shutdown_connections 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 --2- 172.21.15.79:0/2012257475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f14940ff1f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 -- 172.21.15.79:0/2012257475 >> 172.21.15.79:0/2012257475 conn(0x7f14940fa370 msgr2=0x7f14940fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 -- 172.21.15.79:0/2012257475 shutdown_connections 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 -- 172.21.15.79:0/2012257475 wait complete. 2024-06-01T00:46:43.110 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 Processor -- start 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.092+0000 7f149a724640 1 -- start start 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f149a724640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f1494195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f149a724640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1494195960 con 0x7f14940fedd0 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1493fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f1494195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1493fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f1494195420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33010/0 (socket says 172.21.15.79:33010) 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1493fff640 1 -- 172.21.15.79:0/2058819230 learned_addr learned my addr 172.21.15.79:0/2058819230 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1493fff640 1 -- 172.21.15.79:0/2058819230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484009560 con 0x7f14940fedd0 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1493fff640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f1494195420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1484005610 tx=0x7f1484005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1491ffb640 1 -- 172.21.15.79:0/2058819230 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1484004b20 con 0x7f14940fedd0 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1491ffb640 1 -- 172.21.15.79:0/2058819230 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1484004c80 con 0x7f14940fedd0 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1494195b60 con 0x7f14940fedd0 2024-06-01T00:46:43.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f1491ffb640 1 -- 172.21.15.79:0/2058819230 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1484031a90 con 0x7f14940fedd0 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.093+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1494196000 con 0x7f14940fedd0 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.094+0000 7f1491ffb640 1 -- 172.21.15.79:0/2058819230 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f1484031d00 con 0x7f14940fedd0 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.094+0000 7f1491ffb640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f14800378e0 0x7f1480039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.094+0000 7f1491ffb640 1 -- 172.21.15.79:0/2058819230 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f148406b410 con 0x7f14940fedd0 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.094+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1494061ae0 con 0x7f14940fedd0 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.094+0000 7f148bfff640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f14800378e0 0x7f1480039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.095+0000 7f148bfff640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f14800378e0 0x7f1480039da0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f147c0098c0 tx=0x7f147c006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:43.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.099+0000 7f1491ffb640 1 -- 172.21.15.79:0/2058819230 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1484035020 con 0x7f14940fedd0 2024-06-01T00:46:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.278+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f1494060980 con 0x7f14800378e0 2024-06-01T00:46:43.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.281+0000 7f1491ffb640 1 -- 172.21.15.79:0/2058819230 <== mgr.14162 v2:172.21.15.79:6800/3983389754 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1494060980 con 0x7f14800378e0 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f14800378e0 msgr2=0x7f1480039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f14800378e0 0x7f1480039da0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f147c0098c0 tx=0x7f147c006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 msgr2=0x7f1494195420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f1494195420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1484005610 tx=0x7f1484005640 comp rx=0 tx=0).stop 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 shutdown_connections 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f14800378e0 0x7f1480039da0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 --2- 172.21.15.79:0/2058819230 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f14940fedd0 0x7f1494195420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.283+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 >> 172.21.15.79:0/2058819230 conn(0x7f14940fa370 msgr2=0x7f1494068730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.284+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 shutdown_connections 2024-06-01T00:46:43.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:43.284+0000 7f149a724640 1 -- 172.21.15.79:0/2058819230 wait complete. 2024-06-01T00:46:43.929 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2024-06-01T00:46:43.929 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:46:43.929 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2024-06-01T00:46:43.948 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:46:43.948 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-01T00:46:44.006 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2024-06-01T00:46:44.006 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch host add smithi160 2024-06-01T00:46:44.219 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:46:44.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:44.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:44.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:44.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:44.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:44.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:44.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-01T00:46:44.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-01T00:46:44.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:46:44.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:44 smithi079 ceph-mon[29356]: Deploying daemon crash.smithi079 on smithi079 2024-06-01T00:46:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.899+0000 7ff9edaff640 1 -- 172.21.15.79:0/3464450766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 msgr2=0x7ff9e8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.899+0000 7ff9edaff640 1 --2- 172.21.15.79:0/3464450766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8102e00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009880 tx=0x7ff9d802a5d0 comp rx=0 tx=0).stop 2024-06-01T00:46:46.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.899+0000 7ff9edaff640 1 -- 172.21.15.79:0/3464450766 shutdown_connections 2024-06-01T00:46:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.899+0000 7ff9edaff640 1 --2- 172.21.15.79:0/3464450766 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8102e00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.899+0000 7ff9edaff640 1 -- 172.21.15.79:0/3464450766 >> 172.21.15.79:0/3464450766 conn(0x7ff9e80fa380 msgr2=0x7ff9e80fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.899+0000 7ff9edaff640 1 -- 172.21.15.79:0/3464450766 shutdown_connections 2024-06-01T00:46:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.899+0000 7ff9edaff640 1 -- 172.21.15.79:0/3464450766 wait complete. 2024-06-01T00:46:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9edaff640 1 Processor -- start 2024-06-01T00:46:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9edaff640 1 -- start start 2024-06-01T00:46:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9edaff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8100240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9edaff640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8100780 con 0x7ff9e8100a10 2024-06-01T00:46:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8100240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9e77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8100240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33038/0 (socket says 172.21.15.79:33038) 2024-06-01T00:46:46.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9e77fe640 1 -- 172.21.15.79:0/2159173904 learned_addr learned my addr 172.21.15.79:0/2159173904 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:46.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.900+0000 7ff9e77fe640 1 -- 172.21.15.79:0/2159173904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8009560 con 0x7ff9e8100a10 2024-06-01T00:46:46.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.901+0000 7ff9e77fe640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8100240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006010 tx=0x7ff9d8005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:46.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.901+0000 7ff9e4ff9640 1 -- 172.21.15.79:0/2159173904 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d8004aa0 con 0x7ff9e8100a10 2024-06-01T00:46:46.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.901+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e80fe9c0 con 0x7ff9e8100a10 2024-06-01T00:46:46.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.901+0000 7ff9e4ff9640 1 -- 172.21.15.79:0/2159173904 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9d8004c00 con 0x7ff9e8100a10 2024-06-01T00:46:46.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.901+0000 7ff9e4ff9640 1 -- 172.21.15.79:0/2159173904 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d8031a80 con 0x7ff9e8100a10 2024-06-01T00:46:46.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.901+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e80fee60 con 0x7ff9e8100a10 2024-06-01T00:46:46.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.902+0000 7ff9e4ff9640 1 -- 172.21.15.79:0/2159173904 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7ff9d8031d40 con 0x7ff9e8100a10 2024-06-01T00:46:46.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.902+0000 7ff9e4ff9640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7ff9d0037890 0x7ff9d0039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:46.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.902+0000 7ff9e4ff9640 1 -- 172.21.15.79:0/2159173904 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff9d806c3e0 con 0x7ff9e8100a10 2024-06-01T00:46:46.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.902+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e81066a0 con 0x7ff9e8100a10 2024-06-01T00:46:46.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.902+0000 7ff9e6ffd640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7ff9d0037890 0x7ff9d0039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:46.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.903+0000 7ff9e6ffd640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7ff9d0037890 0x7ff9d0039d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098c0 tx=0x7ff9dc006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:46.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:46.907+0000 7ff9e4ff9640 1 -- 172.21.15.79:0/2159173904 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff9d8040080 con 0x7ff9e8100a10 2024-06-01T00:46:47.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:47.103+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9e8061b60 con 0x7ff9d0037890 2024-06-01T00:46:48.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:48.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:48.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:48.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:48.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:48.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:48.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:48.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:48 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:49.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:49 smithi079 ceph-mon[29356]: Deploying daemon node-exporter.smithi079 on smithi079 2024-06-01T00:46:49.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:49 smithi079 ceph-mon[29356]: from='client.14191 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:46:49.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:49 smithi079 ceph-mon[29356]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:49.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:49 smithi079 ceph-mon[29356]: Deploying cephadm binary to smithi160 2024-06-01T00:46:49.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:49 smithi079 ceph-mon[29356]: Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-01T00:46:49.858 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi160' with addr '172.21.15.160' 2024-06-01T00:46:49.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.855+0000 7ff9e4ff9640 1 -- 172.21.15.79:0/2159173904 <== mgr.14162 v2:172.21.15.79:6800/3983389754 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff9e8061b60 con 0x7ff9d0037890 2024-06-01T00:46:49.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7ff9d0037890 msgr2=0x7ff9d0039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:49.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7ff9d0037890 0x7ff9d0039d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098c0 tx=0x7ff9dc006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:49.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 msgr2=0x7ff9e8100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:49.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8100240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006010 tx=0x7ff9d8005630 comp rx=0 tx=0).stop 2024-06-01T00:46:49.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 shutdown_connections 2024-06-01T00:46:49.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7ff9d0037890 0x7ff9d0039d50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:49.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 --2- 172.21.15.79:0/2159173904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff9e8100a10 0x7ff9e8100240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.858+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 >> 172.21.15.79:0/2159173904 conn(0x7ff9e80fa380 msgr2=0x7ff9e80fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.859+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 shutdown_connections 2024-06-01T00:46:49.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:49.859+0000 7ff9edaff640 1 -- 172.21.15.79:0/2159173904 wait complete. 2024-06-01T00:46:51.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:50 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:51.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:50 smithi079 ceph-mon[29356]: Added host smithi160 2024-06-01T00:46:51.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:50 smithi079 ceph-mon[29356]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:51.167 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-01T00:46:51.672 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:46:52.768 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:52 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:52.768 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:52 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:52.768 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:52 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:52.768 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:52 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:52.768 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:52 smithi079 ceph-mon[29356]: Deploying daemon alertmanager.smithi079 on smithi079 2024-06-01T00:46:52.768 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:52 smithi079 ceph-mon[29356]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:52.768 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:52 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:46:53.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.225+0000 7f91347c1640 1 -- 172.21.15.79:0/1709923099 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 msgr2=0x7f912c0ff230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:53.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.225+0000 7f91347c1640 1 --2- 172.21.15.79:0/1709923099 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c0ff230 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9128009880 tx=0x7f912802a5d0 comp rx=0 tx=0).stop 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.225+0000 7f91347c1640 1 -- 172.21.15.79:0/1709923099 shutdown_connections 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.225+0000 7f91347c1640 1 --2- 172.21.15.79:0/1709923099 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c0ff230 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.225+0000 7f91347c1640 1 -- 172.21.15.79:0/1709923099 >> 172.21.15.79:0/1709923099 conn(0x7f912c0fa3b0 msgr2=0x7f912c0fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.225+0000 7f91347c1640 1 -- 172.21.15.79:0/1709923099 shutdown_connections 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.225+0000 7f91347c1640 1 -- 172.21.15.79:0/1709923099 wait complete. 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f91347c1640 1 Processor -- start 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f91347c1640 1 -- start start 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f91347c1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c194330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f91347c1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c194870 con 0x7f912c0fee10 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f913255d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c194330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:53.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f913255d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c194330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43256/0 (socket says 172.21.15.79:43256) 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f913255d640 1 -- 172.21.15.79:0/1368836271 learned_addr learned my addr 172.21.15.79:0/1368836271 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f913255d640 1 -- 172.21.15.79:0/1368836271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128009560 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.226+0000 7f913255d640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c194330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9128009230 tx=0x7f91280056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.227+0000 7f91237fe640 1 -- 172.21.15.79:0/1368836271 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9128004aa0 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.227+0000 7f91237fe640 1 -- 172.21.15.79:0/1368836271 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9128004c00 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.227+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912c194a70 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.227+0000 7f91237fe640 1 -- 172.21.15.79:0/1368836271 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9128031af0 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.227+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912c194f10 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.228+0000 7f91237fe640 1 -- 172.21.15.79:0/1368836271 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f9128031c50 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.228+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912c061ae0 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.228+0000 7f91237fe640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f91180378e0 0x7f9118039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.228+0000 7f91237fe640 1 -- 172.21.15.79:0/1368836271 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f912806b3c0 con 0x7f912c0fee10 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.228+0000 7f9131d5c640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f91180378e0 0x7f9118039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:53.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.228+0000 7f9131d5c640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f91180378e0 0x7f9118039da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f911c0098c0 tx=0x7f911c006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:53.233 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.232+0000 7f91237fe640 1 -- 172.21.15.79:0/1368836271 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9128043360 con 0x7f912c0fee10 2024-06-01T00:46:53.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.415+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f912c02c4a0 con 0x7f91180378e0 2024-06-01T00:46:53.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.416+0000 7f91237fe640 1 -- 172.21.15.79:0/1368836271 <== mgr.14162 v2:172.21.15.79:6800/3983389754 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f912c02c4a0 con 0x7f91180378e0 2024-06-01T00:46:53.418 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:46:53.418 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}] 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f91180378e0 msgr2=0x7f9118039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f91180378e0 0x7f9118039da0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f911c0098c0 tx=0x7f911c006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 msgr2=0x7f912c194330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c194330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9128009230 tx=0x7f91280056e0 comp rx=0 tx=0).stop 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 shutdown_connections 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f91180378e0 0x7f9118039da0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 --2- 172.21.15.79:0/1368836271 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f912c0fee10 0x7f912c194330 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 >> 172.21.15.79:0/1368836271 conn(0x7f912c0fa3b0 msgr2=0x7f912c0faee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 shutdown_connections 2024-06-01T00:46:53.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:53.418+0000 7f91347c1640 1 -- 172.21.15.79:0/1368836271 wait complete. 2024-06-01T00:46:54.400 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-01T00:46:54.400 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-01T00:46:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:46:55.281 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:55 smithi079 ceph-mon[29356]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T00:46:55.281 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:55 smithi079 ceph-mon[29356]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:57.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:57 smithi079 ceph-mon[29356]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:57.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.724+0000 7f9bee15a640 1 -- 172.21.15.79:0/2675329213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 msgr2=0x7f9be80ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.724+0000 7f9bee15a640 1 --2- 172.21.15.79:0/2675329213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be80ff200 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009880 tx=0x7f9bd802a5d0 comp rx=0 tx=0).stop 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.724+0000 7f9bee15a640 1 -- 172.21.15.79:0/2675329213 shutdown_connections 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.724+0000 7f9bee15a640 1 --2- 172.21.15.79:0/2675329213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be80ff200 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.724+0000 7f9bee15a640 1 -- 172.21.15.79:0/2675329213 >> 172.21.15.79:0/2675329213 conn(0x7f9be80fa380 msgr2=0x7f9be80fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.724+0000 7f9bee15a640 1 -- 172.21.15.79:0/2675329213 shutdown_connections 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.724+0000 7f9bee15a640 1 -- 172.21.15.79:0/2675329213 wait complete. 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9bee15a640 1 Processor -- start 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9bee15a640 1 -- start start 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9bee15a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be8068130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9bee15a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be8068670 con 0x7f9be80fede0 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9be77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be8068130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:57.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9be77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be8068130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43274/0 (socket says 172.21.15.79:43274) 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9be77fe640 1 -- 172.21.15.79:0/1979565680 learned_addr learned my addr 172.21.15.79:0/1979565680 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9be77fe640 1 -- 172.21.15.79:0/1979565680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd8009560 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.725+0000 7f9be77fe640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be8068130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8005610 tx=0x7f9bd8005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.726+0000 7f9be57fa640 1 -- 172.21.15.79:0/1979565680 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd8004b20 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.726+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be81980f0 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.726+0000 7f9be57fa640 1 -- 172.21.15.79:0/1979565680 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bd8004c80 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.726+0000 7f9be57fa640 1 -- 172.21.15.79:0/1979565680 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd8031a90 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.726+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be8198590 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.726+0000 7f9be57fa640 1 -- 172.21.15.79:0/1979565680 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f9bd8031d50 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.727+0000 7f9be57fa640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f9bd4037890 0x7f9bd4039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.727+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be8061ae0 con 0x7f9be80fede0 2024-06-01T00:46:57.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.727+0000 7f9be57fa640 1 -- 172.21.15.79:0/1979565680 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9bd806b370 con 0x7f9be80fede0 2024-06-01T00:46:57.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.727+0000 7f9bdffff640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f9bd4037890 0x7f9bd4039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:46:57.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.727+0000 7f9bdffff640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f9bd4037890 0x7f9bd4039d50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00098c0 tx=0x7f9bd0006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:46:57.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.731+0000 7f9be57fa640 1 -- 172.21.15.79:0/1979565680 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9bd8040080 con 0x7f9be80fede0 2024-06-01T00:46:57.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:57.916+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9be818dc00 con 0x7f9be80fede0 2024-06-01T00:46:58.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.080+0000 7f9be57fa640 1 -- 172.21.15.79:0/1979565680 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9bd804ddb0 con 0x7f9be80fede0 2024-06-01T00:46:58.082 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f9bd4037890 msgr2=0x7f9bd4039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f9bd4037890 0x7f9bd4039d50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00098c0 tx=0x7f9bd0006db0 comp rx=0 tx=0).stop 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 msgr2=0x7f9be8068130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be8068130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8005610 tx=0x7f9bd8005640 comp rx=0 tx=0).stop 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 shutdown_connections 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f9bd4037890 0x7f9bd4039d50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 --2- 172.21.15.79:0/1979565680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9be80fede0 0x7f9be8068130 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 >> 172.21.15.79:0/1979565680 conn(0x7f9be80fa380 msgr2=0x7f9be80faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 shutdown_connections 2024-06-01T00:46:58.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:46:58.082+0000 7f9bee15a640 1 -- 172.21.15.79:0/1979565680 wait complete. 2024-06-01T00:46:58.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:58 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1979565680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-01T00:46:59.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:59 smithi079 ceph-mon[29356]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:46:59.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:59 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1979565680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-01T00:46:59.218 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:46:59 smithi079 ceph-mon[29356]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T00:46:59.245 INFO:tasks.cephadm:Adding mon.smithi079 on smithi079 2024-06-01T00:46:59.245 INFO:tasks.cephadm:Adding mon.smithi160 on smithi160 2024-06-01T00:46:59.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch apply mon '2;smithi079:172.21.15.79=smithi079;smithi160:172.21.15.160=smithi160' 2024-06-01T00:46:59.448 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:46:59.495 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.158+0000 7f3957cc9640 1 -- 172.21.15.160:0/2384328465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 msgr2=0x7f39500ff1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.158+0000 7f3957cc9640 1 --2- 172.21.15.160:0/2384328465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f39500ff1f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f394c009880 tx=0x7f394c02a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.158+0000 7f3957cc9640 1 -- 172.21.15.160:0/2384328465 shutdown_connections 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.158+0000 7f3957cc9640 1 --2- 172.21.15.160:0/2384328465 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f39500ff1f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.158+0000 7f3957cc9640 1 -- 172.21.15.160:0/2384328465 >> 172.21.15.160:0/2384328465 conn(0x7f39500fa370 msgr2=0x7f39500fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.159+0000 7f3957cc9640 1 -- 172.21.15.160:0/2384328465 shutdown_connections 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.159+0000 7f3957cc9640 1 -- 172.21.15.160:0/2384328465 wait complete. 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.159+0000 7f3957cc9640 1 Processor -- start 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.159+0000 7f3957cc9640 1 -- start start 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.159+0000 7f3957cc9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f3950068c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:01.161 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.159+0000 7f3957cc9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39500691d0 con 0x7f39500fedd0 2024-06-01T00:47:01.162 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3955a65640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f3950068c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:01.162 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3955a65640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f3950068c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:40072/0 (socket says 172.21.15.160:40072) 2024-06-01T00:47:01.162 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3955a65640 1 -- 172.21.15.160:0/1462509750 learned_addr learned my addr 172.21.15.160:0/1462509750 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:01.163 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3955a65640 1 -- 172.21.15.160:0/1462509750 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c009560 con 0x7f39500fedd0 2024-06-01T00:47:01.163 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3955a65640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f3950068c90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f394c005610 tx=0x7f394c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:01.163 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3946ffd640 1 -- 172.21.15.160:0/1462509750 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c004b20 con 0x7f39500fedd0 2024-06-01T00:47:01.163 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3946ffd640 1 -- 172.21.15.160:0/1462509750 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f394c004c80 con 0x7f39500fedd0 2024-06-01T00:47:01.163 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.160+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3950198190 con 0x7f39500fedd0 2024-06-01T00:47:01.163 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.161+0000 7f3946ffd640 1 -- 172.21.15.160:0/1462509750 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394c031a90 con 0x7f39500fedd0 2024-06-01T00:47:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.161+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39501985d0 con 0x7f39500fedd0 2024-06-01T00:47:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.161+0000 7f3946ffd640 1 -- 172.21.15.160:0/1462509750 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f394c031d50 con 0x7f39500fedd0 2024-06-01T00:47:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.161+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3950061ae0 con 0x7f39500fedd0 2024-06-01T00:47:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.162+0000 7f3946ffd640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f393c037890 0x7f393c039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.162+0000 7f3946ffd640 1 -- 172.21.15.160:0/1462509750 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f394c06b3d0 con 0x7f39500fedd0 2024-06-01T00:47:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.162+0000 7f3955264640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f393c037890 0x7f393c039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:01.164 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.162+0000 7f3955264640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f393c037890 0x7f393c039d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f39400098c0 tx=0x7f3940006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:01.168 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.167+0000 7f3946ffd640 1 -- 172.21.15.160:0/1462509750 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f394c04dce0 con 0x7f39500fedd0 2024-06-01T00:47:01.325 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:01 smithi079 ceph-mon[29356]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:01.347 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.346+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 --> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}) v1 -- 0x7f395002c660 con 0x7f393c037890 2024-06-01T00:47:01.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.351+0000 7f3946ffd640 1 -- 172.21.15.160:0/1462509750 <== mgr.14162 v2:172.21.15.79:6800/3983389754 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f395002c660 con 0x7f393c037890 2024-06-01T00:47:01.353 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2024-06-01T00:47:01.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f393c037890 msgr2=0x7f393c039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:01.354 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f393c037890 0x7f393c039d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f39400098c0 tx=0x7f3940006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 msgr2=0x7f3950068c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f3950068c90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f394c005610 tx=0x7f394c005640 comp rx=0 tx=0).stop 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 shutdown_connections 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f393c037890 0x7f393c039d50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 --2- 172.21.15.160:0/1462509750 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39500fedd0 0x7f3950068c90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 >> 172.21.15.160:0/1462509750 conn(0x7f39500fa370 msgr2=0x7f39500faea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 shutdown_connections 2024-06-01T00:47:01.355 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:01.353+0000 7f3957cc9640 1 -- 172.21.15.160:0/1462509750 wait complete. 2024-06-01T00:47:01.879 DEBUG:teuthology.orchestra.run.smithi160:mon.smithi160> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi160.service 2024-06-01T00:47:01.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:01.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:02.108 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:02.155 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:02.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:47:02.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: Saving service mon spec with placement smithi079:172.21.15.79=smithi079;smithi160:172.21.15.160=smithi160;count:2 2024-06-01T00:47:02.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:02.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T00:47:02.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:02 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:03.497 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.493+0000 7f2b54e55640 1 -- 172.21.15.160:0/716264037 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 msgr2=0x7f2b500ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.493+0000 7f2b54e55640 1 --2- 172.21.15.160:0/716264037 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500ff200 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009880 tx=0x7f2b3802a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.493+0000 7f2b54e55640 1 -- 172.21.15.160:0/716264037 shutdown_connections 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.493+0000 7f2b54e55640 1 --2- 172.21.15.160:0/716264037 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500ff200 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.493+0000 7f2b54e55640 1 -- 172.21.15.160:0/716264037 >> 172.21.15.160:0/716264037 conn(0x7f2b500fa380 msgr2=0x7f2b500fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.493+0000 7f2b54e55640 1 -- 172.21.15.160:0/716264037 shutdown_connections 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.493+0000 7f2b54e55640 1 -- 172.21.15.160:0/716264037 wait complete. 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.494+0000 7f2b54e55640 1 Processor -- start 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.494+0000 7f2b54e55640 1 -- start start 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.494+0000 7f2b54e55640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500680f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:03.498 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.494+0000 7f2b54e55640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b50068630 con 0x7f2b500fede0 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.494+0000 7f2b4e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.494+0000 7f2b4e59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500680f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:40092/0 (socket says 172.21.15.160:40092) 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.494+0000 7f2b4e59c640 1 -- 172.21.15.160:0/4027875508 learned_addr learned my addr 172.21.15.160:0/4027875508 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.495+0000 7f2b4e59c640 1 -- 172.21.15.160:0/4027875508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b38009560 con 0x7f2b500fede0 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.495+0000 7f2b4e59c640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500680f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2b38005610 tx=0x7f2b38005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.495+0000 7f2b477fe640 1 -- 172.21.15.160:0/4027875508 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b38004b20 con 0x7f2b500fede0 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.495+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b50198130 con 0x7f2b500fede0 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.495+0000 7f2b477fe640 1 -- 172.21.15.160:0/4027875508 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b38004c80 con 0x7f2b500fede0 2024-06-01T00:47:03.499 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.495+0000 7f2b477fe640 1 -- 172.21.15.160:0/4027875508 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b38031a90 con 0x7f2b500fede0 2024-06-01T00:47:03.500 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.495+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b501985d0 con 0x7f2b500fede0 2024-06-01T00:47:03.500 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.496+0000 7f2b477fe640 1 -- 172.21.15.160:0/4027875508 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f2b38031d50 con 0x7f2b500fede0 2024-06-01T00:47:03.500 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.496+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b50061ae0 con 0x7f2b500fede0 2024-06-01T00:47:03.500 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.496+0000 7f2b477fe640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2b3c037890 0x7f2b3c039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:03.500 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.496+0000 7f2b477fe640 1 -- 172.21.15.160:0/4027875508 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2b3806b340 con 0x7f2b500fede0 2024-06-01T00:47:03.500 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.496+0000 7f2b4dd9b640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2b3c037890 0x7f2b3c039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:03.500 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.497+0000 7f2b4dd9b640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2b3c037890 0x7f2b3c039d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2b400098c0 tx=0x7f2b40006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:03.502 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.501+0000 7f2b477fe640 1 -- 172.21.15.160:0/4027875508 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2b380443b0 con 0x7f2b500fede0 2024-06-01T00:47:03.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:03 smithi079 ceph-mon[29356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T00:47:03.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:03 smithi079 ceph-mon[29356]: Deploying daemon grafana.smithi079 on smithi079 2024-06-01T00:47:03.738 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.737+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b5018dc00 con 0x7f2b500fede0 2024-06-01T00:47:03.739 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.738+0000 7f2b477fe640 1 -- 172.21.15.160:0/4027875508 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2b38035070 con 0x7f2b500fede0 2024-06-01T00:47:03.739 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:03.739 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:03.740 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:03.741 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2b3c037890 msgr2=0x7f2b3c039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:03.742 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2b3c037890 0x7f2b3c039d50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2b400098c0 tx=0x7f2b40006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:03.742 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 msgr2=0x7f2b500680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:03.742 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500680f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2b38005610 tx=0x7f2b38005640 comp rx=0 tx=0).stop 2024-06-01T00:47:03.742 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 shutdown_connections 2024-06-01T00:47:03.742 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f2b3c037890 0x7f2b3c039d50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:03.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 --2- 172.21.15.160:0/4027875508 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2b500fede0 0x7f2b500680f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:03.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 >> 172.21.15.160:0/4027875508 conn(0x7f2b500fa380 msgr2=0x7f2b500faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:03.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 shutdown_connections 2024-06-01T00:47:03.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:03.740+0000 7f2b54e55640 1 -- 172.21.15.160:0/4027875508 wait complete. 2024-06-01T00:47:04.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:04 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/4027875508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:04.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:04 smithi079 ceph-mon[29356]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:05.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:05.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:05.519 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:05.565 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 -- 172.21.15.160:0/383878299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 msgr2=0x7fdcf00ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 --2- 172.21.15.160:0/383878299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00ff200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009880 tx=0x7fdce002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 -- 172.21.15.160:0/383878299 shutdown_connections 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 --2- 172.21.15.160:0/383878299 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00ff200 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 -- 172.21.15.160:0/383878299 >> 172.21.15.160:0/383878299 conn(0x7fdcf00fa3a0 msgr2=0x7fdcf00fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 -- 172.21.15.160:0/383878299 shutdown_connections 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 -- 172.21.15.160:0/383878299 wait complete. 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.962+0000 7fdcf66c7640 1 Processor -- start 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdcf66c7640 1 -- start start 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdcf66c7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:06.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdcf66c7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf00fc700 con 0x7fdcf00fede0 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdceffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdceffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00fc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:40108/0 (socket says 172.21.15.160:40108) 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdceffff640 1 -- 172.21.15.160:0/1707632200 learned_addr learned my addr 172.21.15.160:0/1707632200 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdceffff640 1 -- 172.21.15.160:0/1707632200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce0009560 con 0x7fdcf00fede0 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.963+0000 7fdceffff640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00fc1c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009230 tx=0x7fdce00056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.964+0000 7fdced7fa640 1 -- 172.21.15.160:0/1707632200 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdce0004aa0 con 0x7fdcf00fede0 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.964+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf00fa940 con 0x7fdcf00fede0 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.964+0000 7fdced7fa640 1 -- 172.21.15.160:0/1707632200 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdce0004c00 con 0x7fdcf00fede0 2024-06-01T00:47:06.966 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.964+0000 7fdced7fa640 1 -- 172.21.15.160:0/1707632200 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdce0031a60 con 0x7fdcf00fede0 2024-06-01T00:47:06.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.964+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf00fade0 con 0x7fdcf00fede0 2024-06-01T00:47:06.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.965+0000 7fdced7fa640 1 -- 172.21.15.160:0/1707632200 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fdce0031bc0 con 0x7fdcf00fede0 2024-06-01T00:47:06.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.965+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf01055c0 con 0x7fdcf00fede0 2024-06-01T00:47:06.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.965+0000 7fdced7fa640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fdcdc0378e0 0x7fdcdc039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:06.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.965+0000 7fdced7fa640 1 -- 172.21.15.160:0/1707632200 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdce006b2d0 con 0x7fdcf00fede0 2024-06-01T00:47:06.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.965+0000 7fdcef7fe640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fdcdc0378e0 0x7fdcdc039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:06.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.965+0000 7fdcef7fe640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fdcdc0378e0 0x7fdcdc039da0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdcd80098c0 tx=0x7fdcd8006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:06.975 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:06.974+0000 7fdced7fa640 1 -- 172.21.15.160:0/1707632200 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fdce004d7c0 con 0x7fdcf00fede0 2024-06-01T00:47:07.215 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.214+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcf018e9c0 con 0x7fdcf00fede0 2024-06-01T00:47:07.216 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.214+0000 7fdced7fa640 1 -- 172.21.15.160:0/1707632200 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fdce0035020 con 0x7fdcf00fede0 2024-06-01T00:47:07.216 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:07.216 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:07.216 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fdcdc0378e0 msgr2=0x7fdcdc039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fdcdc0378e0 0x7fdcdc039da0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdcd80098c0 tx=0x7fdcd8006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 msgr2=0x7fdcf00fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00fc1c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009230 tx=0x7fdce00056e0 comp rx=0 tx=0).stop 2024-06-01T00:47:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 shutdown_connections 2024-06-01T00:47:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fdcdc0378e0 0x7fdcdc039da0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:07.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 --2- 172.21.15.160:0/1707632200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdcf00fede0 0x7fdcf00fc1c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:07.219 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 >> 172.21.15.160:0/1707632200 conn(0x7fdcf00fa3a0 msgr2=0x7fdcf0068790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:07.219 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 shutdown_connections 2024-06-01T00:47:07.219 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:07.217+0000 7fdcf66c7640 1 -- 172.21.15.160:0/1707632200 wait complete. 2024-06-01T00:47:07.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:07 smithi079 ceph-mon[29356]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:08.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:08 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:08.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:08 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1707632200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:08.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:08.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:09.021 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:09.068 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:09.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:09 smithi079 ceph-mon[29356]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:10.504 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1687453936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 msgr2=0x7fb394102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:10.504 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 --2- 172.21.15.160:0/1687453936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394102df0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb390009880 tx=0x7fb39002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:10.504 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1687453936 shutdown_connections 2024-06-01T00:47:10.505 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 --2- 172.21.15.160:0/1687453936 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394102df0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:10.505 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1687453936 >> 172.21.15.160:0/1687453936 conn(0x7fb3940fa370 msgr2=0x7fb3940fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:10.505 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1687453936 shutdown_connections 2024-06-01T00:47:10.505 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1687453936 wait complete. 2024-06-01T00:47:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.492+0000 7fb39b5ca640 1 Processor -- start 2024-06-01T00:47:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb39b5ca640 1 -- start start 2024-06-01T00:47:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb39b5ca640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb39b5ca640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3941958b0 con 0x7fb394100a00 2024-06-01T00:47:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb399366640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb399366640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394195370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:33996/0 (socket says 172.21.15.160:33996) 2024-06-01T00:47:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb399366640 1 -- 172.21.15.160:0/1002713913 learned_addr learned my addr 172.21.15.160:0/1002713913 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:10.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb399366640 1 -- 172.21.15.160:0/1002713913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb390009560 con 0x7fb394100a00 2024-06-01T00:47:10.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb399366640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394195370 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb390006010 tx=0x7fb390005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:10.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb38a7fc640 1 -- 172.21.15.160:0/1002713913 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb390004aa0 con 0x7fb394100a00 2024-06-01T00:47:10.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb394195ab0 con 0x7fb394100a00 2024-06-01T00:47:10.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb38a7fc640 1 -- 172.21.15.160:0/1002713913 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb390004c00 con 0x7fb394100a00 2024-06-01T00:47:10.507 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.493+0000 7fb38a7fc640 1 -- 172.21.15.160:0/1002713913 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb390031a80 con 0x7fb394100a00 2024-06-01T00:47:10.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.494+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb394195f50 con 0x7fb394100a00 2024-06-01T00:47:10.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.494+0000 7fb38a7fc640 1 -- 172.21.15.160:0/1002713913 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fb390031cf0 con 0x7fb394100a00 2024-06-01T00:47:10.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.494+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb394061ae0 con 0x7fb394100a00 2024-06-01T00:47:10.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.495+0000 7fb38a7fc640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb3800378e0 0x7fb380039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:10.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.495+0000 7fb38a7fc640 1 -- 172.21.15.160:0/1002713913 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb39006b430 con 0x7fb394100a00 2024-06-01T00:47:10.508 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.495+0000 7fb398b65640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb3800378e0 0x7fb380039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:10.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.495+0000 7fb398b65640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb3800378e0 0x7fb380039da0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb3840098c0 tx=0x7fb384006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:10.509 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.500+0000 7fb38a7fc640 1 -- 172.21.15.160:0/1002713913 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb39004d9a0 con 0x7fb394100a00 2024-06-01T00:47:10.740 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.739+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb39402c3e0 con 0x7fb394100a00 2024-06-01T00:47:10.741 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.740+0000 7fb38a7fc640 1 -- 172.21.15.160:0/1002713913 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb390035020 con 0x7fb394100a00 2024-06-01T00:47:10.741 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:10.741 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:10.741 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:10.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb3800378e0 msgr2=0x7fb380039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:10.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb3800378e0 0x7fb380039da0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb3840098c0 tx=0x7fb384006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:10.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 msgr2=0x7fb394195370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:10.743 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394195370 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb390006010 tx=0x7fb390005630 comp rx=0 tx=0).stop 2024-06-01T00:47:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 shutdown_connections 2024-06-01T00:47:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fb3800378e0 0x7fb380039da0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 --2- 172.21.15.160:0/1002713913 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb394100a00 0x7fb394195370 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 >> 172.21.15.160:0/1002713913 conn(0x7fb3940fa370 msgr2=0x7fb3940fada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 shutdown_connections 2024-06-01T00:47:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:10.742+0000 7fb39b5ca640 1 -- 172.21.15.160:0/1002713913 wait complete. 2024-06-01T00:47:11.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:11 smithi079 ceph-mon[29356]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:11.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:11 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1002713913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:12.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:12.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:12.605 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:12.652 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:13.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:13 smithi079 ceph-mon[29356]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.962+0000 7f58269ca640 1 -- 172.21.15.160:0/1393238374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 msgr2=0x7f58200ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.962+0000 7f58269ca640 1 --2- 172.21.15.160:0/1393238374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f58200ff200 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5810009880 tx=0x7f581002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:13.964 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.962+0000 7f58269ca640 1 -- 172.21.15.160:0/1393238374 shutdown_connections 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.962+0000 7f58269ca640 1 --2- 172.21.15.160:0/1393238374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f58200ff200 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.962+0000 7f58269ca640 1 -- 172.21.15.160:0/1393238374 >> 172.21.15.160:0/1393238374 conn(0x7f58200fa3a0 msgr2=0x7f58200fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.962+0000 7f58269ca640 1 -- 172.21.15.160:0/1393238374 shutdown_connections 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.962+0000 7f58269ca640 1 -- 172.21.15.160:0/1393238374 wait complete. 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.963+0000 7f58269ca640 1 Processor -- start 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.963+0000 7f58269ca640 1 -- start start 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.963+0000 7f58269ca640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f5820068d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:13.965 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.963+0000 7f58269ca640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5820069240 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f5820068d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f5820068d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:34004/0 (socket says 172.21.15.160:34004) 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581ffff640 1 -- 172.21.15.160:0/1557245373 learned_addr learned my addr 172.21.15.160:0/1557245373 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581ffff640 1 -- 172.21.15.160:0/1557245373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5810009560 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581ffff640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f5820068d00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5810009230 tx=0x7f58100056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581d7fa640 1 -- 172.21.15.160:0/1557245373 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5810004aa0 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58201981d0 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581d7fa640 1 -- 172.21.15.160:0/1557245373 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5810004c00 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.964+0000 7f581d7fa640 1 -- 172.21.15.160:0/1557245373 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5810031a60 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.965+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5820198610 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.965+0000 7f581d7fa640 1 -- 172.21.15.160:0/1557245373 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f5810031bc0 con 0x7f58200fede0 2024-06-01T00:47:13.967 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.965+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5820061ae0 con 0x7f58200fede0 2024-06-01T00:47:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.966+0000 7f581d7fa640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5808037890 0x7f5808039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.966+0000 7f581d7fa640 1 -- 172.21.15.160:0/1557245373 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f581006b320 con 0x7f58200fede0 2024-06-01T00:47:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.966+0000 7f581f7fe640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5808037890 0x7f5808039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:13.968 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.966+0000 7f581f7fe640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5808037890 0x7f5808039d50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58140098c0 tx=0x7f5814006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:13.972 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:13.971+0000 7f581d7fa640 1 -- 172.21.15.160:0/1557245373 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f581004d7c0 con 0x7f58200fede0 2024-06-01T00:47:14.214 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.213+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f582018dc00 con 0x7f58200fede0 2024-06-01T00:47:14.215 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.214+0000 7f581d7fa640 1 -- 172.21.15.160:0/1557245373 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f5810035020 con 0x7f58200fede0 2024-06-01T00:47:14.215 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:14.215 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:14.216 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.216+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5808037890 msgr2=0x7f5808039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.216+0000 7f58269ca640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5808037890 0x7f5808039d50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f58140098c0 tx=0x7f5814006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.216+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 msgr2=0x7f5820068d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.216+0000 7f58269ca640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f5820068d00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5810009230 tx=0x7f58100056e0 comp rx=0 tx=0).stop 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.217+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 shutdown_connections 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.217+0000 7f58269ca640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5808037890 0x7f5808039d50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.217+0000 7f58269ca640 1 --2- 172.21.15.160:0/1557245373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f58200fede0 0x7f5820068d00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.217+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 >> 172.21.15.160:0/1557245373 conn(0x7f58200fa3a0 msgr2=0x7f58200faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:14.218 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.217+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 shutdown_connections 2024-06-01T00:47:14.219 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:14.217+0000 7f58269ca640 1 -- 172.21.15.160:0/1557245373 wait complete. 2024-06-01T00:47:15.442 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:15 smithi079 ceph-mon[29356]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:15.443 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:15 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1557245373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:15.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:15.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:15.968 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:16.015 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:17.355 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:17 smithi079 ceph-mon[29356]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:17.367 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.365+0000 7fad83c1d640 1 -- 172.21.15.160:0/3316783033 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 msgr2=0x7fad7c0ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:17.367 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.365+0000 7fad83c1d640 1 --2- 172.21.15.160:0/3316783033 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c0ff200 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fad78009880 tx=0x7fad7802a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:17.367 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.365+0000 7fad83c1d640 1 -- 172.21.15.160:0/3316783033 shutdown_connections 2024-06-01T00:47:17.367 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.365+0000 7fad83c1d640 1 --2- 172.21.15.160:0/3316783033 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c0ff200 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:17.367 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.365+0000 7fad83c1d640 1 -- 172.21.15.160:0/3316783033 >> 172.21.15.160:0/3316783033 conn(0x7fad7c0fa380 msgr2=0x7fad7c0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:17.367 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.365+0000 7fad83c1d640 1 -- 172.21.15.160:0/3316783033 shutdown_connections 2024-06-01T00:47:17.367 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.365+0000 7fad83c1d640 1 -- 172.21.15.160:0/3316783033 wait complete. 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad83c1d640 1 Processor -- start 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad83c1d640 1 -- start start 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad83c1d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad83c1d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c195990 con 0x7fad7c0fede0 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad819b9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad819b9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c195450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:34010/0 (socket says 172.21.15.160:34010) 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad819b9640 1 -- 172.21.15.160:0/550060646 learned_addr learned my addr 172.21.15.160:0/550060646 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:17.368 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad819b9640 1 -- 172.21.15.160:0/550060646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78009560 con 0x7fad7c0fede0 2024-06-01T00:47:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.366+0000 7fad819b9640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c195450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fad78005610 tx=0x7fad78005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.367+0000 7fad72ffd640 1 -- 172.21.15.160:0/550060646 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad78004b20 con 0x7fad7c0fede0 2024-06-01T00:47:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.367+0000 7fad72ffd640 1 -- 172.21.15.160:0/550060646 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad78004c80 con 0x7fad7c0fede0 2024-06-01T00:47:17.369 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.367+0000 7fad72ffd640 1 -- 172.21.15.160:0/550060646 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad78031a90 con 0x7fad7c0fede0 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.367+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c195b90 con 0x7fad7c0fede0 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.367+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c196030 con 0x7fad7c0fede0 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.368+0000 7fad72ffd640 1 -- 172.21.15.160:0/550060646 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fad78031d00 con 0x7fad7c0fede0 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.368+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7c061ae0 con 0x7fad7c0fede0 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.368+0000 7fad72ffd640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fad680378e0 0x7fad68039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.368+0000 7fad72ffd640 1 -- 172.21.15.160:0/550060646 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fad7806b3c0 con 0x7fad7c0fede0 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.368+0000 7fad811b8640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fad680378e0 0x7fad68039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:17.370 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.369+0000 7fad811b8640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fad680378e0 0x7fad68039da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad6c0098c0 tx=0x7fad6c006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:17.377 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.376+0000 7fad72ffd640 1 -- 172.21.15.160:0/550060646 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fad7804dd20 con 0x7fad7c0fede0 2024-06-01T00:47:17.613 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.612+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad7c02c3e0 con 0x7fad7c0fede0 2024-06-01T00:47:17.613 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.612+0000 7fad72ffd640 1 -- 172.21.15.160:0/550060646 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fad78035070 con 0x7fad7c0fede0 2024-06-01T00:47:17.614 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:17.614 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:17.614 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fad680378e0 msgr2=0x7fad68039da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fad680378e0 0x7fad68039da0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad6c0098c0 tx=0x7fad6c006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 msgr2=0x7fad7c195450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c195450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fad78005610 tx=0x7fad78005640 comp rx=0 tx=0).stop 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 shutdown_connections 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fad680378e0 0x7fad68039da0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 --2- 172.21.15.160:0/550060646 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fad7c0fede0 0x7fad7c195450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 >> 172.21.15.160:0/550060646 conn(0x7fad7c0fa380 msgr2=0x7fad7c067b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 shutdown_connections 2024-06-01T00:47:17.616 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:17.614+0000 7fad83c1d640 1 -- 172.21.15.160:0/550060646 wait complete. 2024-06-01T00:47:18.728 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.728 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/550060646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: Deploying daemon prometheus.smithi079 on smithi079 2024-06-01T00:47:18.729 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:18 smithi079 ceph-mon[29356]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:19.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:19.202 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:19.404 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:19.452 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:20.799 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 -- 172.21.15.160:0/3167030649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 msgr2=0x7fc230102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 --2- 172.21.15.160:0/3167030649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230102d80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc220009880 tx=0x7fc22002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 -- 172.21.15.160:0/3167030649 shutdown_connections 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 --2- 172.21.15.160:0/3167030649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230102d80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 -- 172.21.15.160:0/3167030649 >> 172.21.15.160:0/3167030649 conn(0x7fc2300fa3a0 msgr2=0x7fc2300fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 -- 172.21.15.160:0/3167030649 shutdown_connections 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 -- 172.21.15.160:0/3167030649 wait complete. 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 Processor -- start 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.795+0000 7fc235e9a640 1 -- start start 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.796+0000 7fc235e9a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:20.800 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.796+0000 7fc235e9a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc230195960 con 0x7fc230100990 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.796+0000 7fc22f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.796+0000 7fc22f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230195420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:42178/0 (socket says 172.21.15.160:42178) 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.796+0000 7fc22f7fe640 1 -- 172.21.15.160:0/1395763300 learned_addr learned my addr 172.21.15.160:0/1395763300 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.796+0000 7fc22f7fe640 1 -- 172.21.15.160:0/1395763300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc220009560 con 0x7fc230100990 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.796+0000 7fc22f7fe640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230195420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc220006010 tx=0x7fc220005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.797+0000 7fc22cff9640 1 -- 172.21.15.160:0/1395763300 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc220004aa0 con 0x7fc230100990 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.797+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc230195b60 con 0x7fc230100990 2024-06-01T00:47:20.801 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.797+0000 7fc22cff9640 1 -- 172.21.15.160:0/1395763300 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc220004c00 con 0x7fc230100990 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.797+0000 7fc22cff9640 1 -- 172.21.15.160:0/1395763300 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc220031a80 con 0x7fc230100990 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.797+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc230196000 con 0x7fc230100990 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.797+0000 7fc22cff9640 1 -- 172.21.15.160:0/1395763300 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fc220031d40 con 0x7fc230100990 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.798+0000 7fc22cff9640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc218037890 0x7fc218039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.798+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc230061ae0 con 0x7fc230100990 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.798+0000 7fc22cff9640 1 -- 172.21.15.160:0/1395763300 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc22006b460 con 0x7fc230100990 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.798+0000 7fc22effd640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc218037890 0x7fc218039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:20.802 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.798+0000 7fc22effd640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc218037890 0x7fc218039d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc2240098c0 tx=0x7fc224006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:20.804 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:20.803+0000 7fc22cff9640 1 -- 172.21.15.160:0/1395763300 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc220043360 con 0x7fc230100990 2024-06-01T00:47:21.041 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.039+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc23002c3e0 con 0x7fc230100990 2024-06-01T00:47:21.041 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.040+0000 7fc22cff9640 1 -- 172.21.15.160:0/1395763300 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc220040020 con 0x7fc230100990 2024-06-01T00:47:21.041 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:21.041 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:21.041 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc218037890 msgr2=0x7fc218039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc218037890 0x7fc218039d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc2240098c0 tx=0x7fc224006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 msgr2=0x7fc230195420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230195420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc220006010 tx=0x7fc220005630 comp rx=0 tx=0).stop 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 shutdown_connections 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc218037890 0x7fc218039d50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 --2- 172.21.15.160:0/1395763300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc230100990 0x7fc230195420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 >> 172.21.15.160:0/1395763300 conn(0x7fc2300fa3a0 msgr2=0x7fc2300fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 shutdown_connections 2024-06-01T00:47:21.044 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:21.042+0000 7fc235e9a640 1 -- 172.21.15.160:0/1395763300 wait complete. 2024-06-01T00:47:21.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:21 smithi079 ceph-mon[29356]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:21.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:21 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1395763300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:22.556 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:22.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:22.756 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:22.804 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:23.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:23 smithi079 ceph-mon[29356]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:23.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:23 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.279+0000 7fd63195a640 1 -- 172.21.15.160:0/1430404627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 msgr2=0x7fd62c0ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.279+0000 7fd63195a640 1 --2- 172.21.15.160:0/1430404627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0ff200 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009880 tx=0x7fd61c02a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.279+0000 7fd63195a640 1 -- 172.21.15.160:0/1430404627 shutdown_connections 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.279+0000 7fd63195a640 1 --2- 172.21.15.160:0/1430404627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0ff200 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.279+0000 7fd63195a640 1 -- 172.21.15.160:0/1430404627 >> 172.21.15.160:0/1430404627 conn(0x7fd62c0fa380 msgr2=0x7fd62c0fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.279+0000 7fd63195a640 1 -- 172.21.15.160:0/1430404627 shutdown_connections 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.279+0000 7fd63195a640 1 -- 172.21.15.160:0/1430404627 wait complete. 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.280+0000 7fd63195a640 1 Processor -- start 2024-06-01T00:47:24.284 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.280+0000 7fd63195a640 1 -- start start 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.280+0000 7fd63195a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0680d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.280+0000 7fd63195a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c068610 con 0x7fd62c0fede0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.280+0000 7fd62affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0680d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.280+0000 7fd62affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0680d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:42192/0 (socket says 172.21.15.160:42192) 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.280+0000 7fd62affd640 1 -- 172.21.15.160:0/3837092344 learned_addr learned my addr 172.21.15.160:0/3837092344 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.281+0000 7fd62affd640 1 -- 172.21.15.160:0/3837092344 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c009560 con 0x7fd62c0fede0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.281+0000 7fd62affd640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0680d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd61c005610 tx=0x7fd61c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.281+0000 7fd630958640 1 -- 172.21.15.160:0/3837092344 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd61c004b20 con 0x7fd62c0fede0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.281+0000 7fd630958640 1 -- 172.21.15.160:0/3837092344 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd61c004c80 con 0x7fd62c0fede0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.281+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c198150 con 0x7fd62c0fede0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.281+0000 7fd630958640 1 -- 172.21.15.160:0/3837092344 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd61c031a90 con 0x7fd62c0fede0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.281+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c198590 con 0x7fd62c0fede0 2024-06-01T00:47:24.285 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.282+0000 7fd630958640 1 -- 172.21.15.160:0/3837092344 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7fd61c031d10 con 0x7fd62c0fede0 2024-06-01T00:47:24.286 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.282+0000 7fd630958640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fd618037890 0x7fd618039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:24.286 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.282+0000 7fd630958640 1 -- 172.21.15.160:0/3837092344 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd61c06b470 con 0x7fd62c0fede0 2024-06-01T00:47:24.286 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.282+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62c061ae0 con 0x7fd62c0fede0 2024-06-01T00:47:24.286 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.282+0000 7fd6227fc640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fd618037890 0x7fd618039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:24.286 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.283+0000 7fd6227fc640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fd618037890 0x7fd618039d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6140098c0 tx=0x7fd614006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:24.288 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.287+0000 7fd630958640 1 -- 172.21.15.160:0/3837092344 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd61c035080 con 0x7fd62c0fede0 2024-06-01T00:47:24.529 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.528+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd62c18dc00 con 0x7fd62c0fede0 2024-06-01T00:47:24.530 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.528+0000 7fd630958640 1 -- 172.21.15.160:0/3837092344 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fd61c040020 con 0x7fd62c0fede0 2024-06-01T00:47:24.530 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:24.530 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:24.530 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fd618037890 msgr2=0x7fd618039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:24.531 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fd618037890 0x7fd618039d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd6140098c0 tx=0x7fd614006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 msgr2=0x7fd62c0680d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0680d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd61c005610 tx=0x7fd61c005640 comp rx=0 tx=0).stop 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 shutdown_connections 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fd618037890 0x7fd618039d50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 --2- 172.21.15.160:0/3837092344 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd62c0fede0 0x7fd62c0680d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 >> 172.21.15.160:0/3837092344 conn(0x7fd62c0fa380 msgr2=0x7fd62c0faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 shutdown_connections 2024-06-01T00:47:24.532 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:24.530+0000 7fd63195a640 1 -- 172.21.15.160:0/3837092344 wait complete. 2024-06-01T00:47:25.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:25 smithi079 ceph-mon[29356]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:25.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:25 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/3837092344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:26.086 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:26.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:26.289 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:26.336 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:27.459 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:27 smithi079 ceph-mon[29356]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:27.671 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.667+0000 7f379d5d0640 1 -- 172.21.15.160:0/3415030772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 msgr2=0x7f37980ff180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.667+0000 7f379d5d0640 1 --2- 172.21.15.160:0/3415030772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980ff180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3780009880 tx=0x7f378002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.667+0000 7f379d5d0640 1 -- 172.21.15.160:0/3415030772 shutdown_connections 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.667+0000 7f379d5d0640 1 --2- 172.21.15.160:0/3415030772 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980ff180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.667+0000 7f379d5d0640 1 -- 172.21.15.160:0/3415030772 >> 172.21.15.160:0/3415030772 conn(0x7f37980fa300 msgr2=0x7f37980fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.667+0000 7f379d5d0640 1 -- 172.21.15.160:0/3415030772 shutdown_connections 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.667+0000 7f379d5d0640 1 -- 172.21.15.160:0/3415030772 wait complete. 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.668+0000 7f379d5d0640 1 Processor -- start 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.668+0000 7f379d5d0640 1 -- start start 2024-06-01T00:47:27.672 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.668+0000 7f379d5d0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.668+0000 7f379d5d0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37980fc6b0 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.668+0000 7f3796ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f3796ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980fc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:42204/0 (socket says 172.21.15.160:42204) 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f3796ffd640 1 -- 172.21.15.160:0/2586726844 learned_addr learned my addr 172.21.15.160:0/2586726844 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f3796ffd640 1 -- 172.21.15.160:0/2586726844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780009560 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f3796ffd640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980fc170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3780005610 tx=0x7f3780005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f378ffff640 1 -- 172.21.15.160:0/2586726844 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3780004b20 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37980fa8f0 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f378ffff640 1 -- 172.21.15.160:0/2586726844 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3780004c80 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f378ffff640 1 -- 172.21.15.160:0/2586726844 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3780031a90 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.669+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37980fad90 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.670+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3798105540 con 0x7f37980fed60 2024-06-01T00:47:27.673 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.670+0000 7f378ffff640 1 -- 172.21.15.160:0/2586726844 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 44922+0+0 (secure 0 0 0) 0x7f3780031d50 con 0x7f37980fed60 2024-06-01T00:47:27.674 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.670+0000 7f378ffff640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f3784037890 0x7f3784039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:27.674 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.670+0000 7f378ffff640 1 -- 172.21.15.160:0/2586726844 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f378006b390 con 0x7f37980fed60 2024-06-01T00:47:27.674 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.671+0000 7f37967fc640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f3784037890 0x7f3784039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:27.674 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.671+0000 7f37967fc640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f3784037890 0x7f3784039d50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f37880098c0 tx=0x7f3788006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:27.676 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.676+0000 7f378ffff640 1 -- 172.21.15.160:0/2586726844 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f378004dce0 con 0x7f37980fed60 2024-06-01T00:47:27.918 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.917+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f379818e890 con 0x7f37980fed60 2024-06-01T00:47:27.919 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.917+0000 7f378ffff640 1 -- 172.21.15.160:0/2586726844 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f3780035070 con 0x7f37980fed60 2024-06-01T00:47:27.919 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:27.919 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:27.920 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f3784037890 msgr2=0x7f3784039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f3784037890 0x7f3784039d50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f37880098c0 tx=0x7f3788006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 msgr2=0x7f37980fc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980fc170 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3780005610 tx=0x7f3780005640 comp rx=0 tx=0).stop 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 shutdown_connections 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f3784037890 0x7f3784039d50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 --2- 172.21.15.160:0/2586726844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f37980fed60 0x7f37980fc170 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.919+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 >> 172.21.15.160:0/2586726844 conn(0x7f37980fa300 msgr2=0x7f37980686d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.920+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 shutdown_connections 2024-06-01T00:47:27.921 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:27.920+0000 7f379d5d0640 1 -- 172.21.15.160:0/2586726844 wait complete. 2024-06-01T00:47:28.205 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:28 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/2586726844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:29.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:29 smithi079 ceph-mon[29356]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:29.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:29.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:29.566 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:29.613 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:30.714 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:30 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:30.715 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:30 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:30.715 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:30 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:47:30.715 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:30 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-01T00:47:30.715 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:30 smithi079 ceph-mon[29356]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:31.007 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.004+0000 7fc31cad1640 1 -- 172.21.15.160:0/2266252612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 msgr2=0x7fc3180ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:31.007 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.004+0000 7fc31cad1640 1 --2- 172.21.15.160:0/2266252612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180ff200 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009880 tx=0x7fc30c02a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.004+0000 7fc31cad1640 1 -- 172.21.15.160:0/2266252612 shutdown_connections 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.004+0000 7fc31cad1640 1 --2- 172.21.15.160:0/2266252612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180ff200 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.004+0000 7fc31cad1640 1 -- 172.21.15.160:0/2266252612 >> 172.21.15.160:0/2266252612 conn(0x7fc3180fa380 msgr2=0x7fc3180fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.004+0000 7fc31cad1640 1 -- 172.21.15.160:0/2266252612 shutdown_connections 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.004+0000 7fc31cad1640 1 -- 172.21.15.160:0/2266252612 wait complete. 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.005+0000 7fc31cad1640 1 Processor -- start 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.005+0000 7fc31cad1640 1 -- start start 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.005+0000 7fc31cad1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180680d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.005+0000 7fc31cad1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc318068610 con 0x7fc3180fede0 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.005+0000 7fc31659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180680d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.005+0000 7fc31659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180680d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:46508/0 (socket says 172.21.15.160:46508) 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.005+0000 7fc31659c640 1 -- 172.21.15.160:0/3862849904 learned_addr learned my addr 172.21.15.160:0/3862849904 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:31.008 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.006+0000 7fc31659c640 1 -- 172.21.15.160:0/3862849904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30c009560 con 0x7fc3180fede0 2024-06-01T00:47:31.009 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.006+0000 7fc31659c640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180680d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc30c005610 tx=0x7fc30c005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:31.009 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.006+0000 7fc3077fe640 1 -- 172.21.15.160:0/3862849904 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc30c004b20 con 0x7fc3180fede0 2024-06-01T00:47:31.009 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.006+0000 7fc3077fe640 1 -- 172.21.15.160:0/3862849904 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc30c004c80 con 0x7fc3180fede0 2024-06-01T00:47:31.009 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.006+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc318198150 con 0x7fc3180fede0 2024-06-01T00:47:31.009 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.006+0000 7fc3077fe640 1 -- 172.21.15.160:0/3862849904 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc30c031a90 con 0x7fc3180fede0 2024-06-01T00:47:31.009 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.006+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc318198590 con 0x7fc3180fede0 2024-06-01T00:47:31.010 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.007+0000 7fc3077fe640 1 -- 172.21.15.160:0/3862849904 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 14) v1 ==== 44936+0+0 (secure 0 0 0) 0x7fc30c031d50 con 0x7fc3180fede0 2024-06-01T00:47:31.010 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.007+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc318061ae0 con 0x7fc3180fede0 2024-06-01T00:47:31.010 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.007+0000 7fc3077fe640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 0x7fc300039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:31.010 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.007+0000 7fc3077fe640 1 -- 172.21.15.160:0/3862849904 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc30c06b580 con 0x7fc3180fede0 2024-06-01T00:47:31.010 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.007+0000 7fc315d9b640 1 -- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 msgr2=0x7fc300039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3983389754 2024-06-01T00:47:31.010 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.007+0000 7fc315d9b640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 0x7fc300039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-01T00:47:31.013 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.012+0000 7fc3077fe640 1 -- 172.21.15.160:0/3862849904 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc30c04db40 con 0x7fc3180fede0 2024-06-01T00:47:31.209 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.208+0000 7fc315d9b640 1 -- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 msgr2=0x7fc300039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3983389754 2024-06-01T00:47:31.209 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.208+0000 7fc315d9b640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 0x7fc300039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-01T00:47:31.252 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.251+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc31818dc00 con 0x7fc3180fede0 2024-06-01T00:47:31.252 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.251+0000 7fc3077fe640 1 -- 172.21.15.160:0/3862849904 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc30c035070 con 0x7fc3180fede0 2024-06-01T00:47:31.252 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:31.253 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:31.253 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:31.254 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 msgr2=0x7fc300039da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-01T00:47:31.254 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 0x7fc300039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 msgr2=0x7fc3180680d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180680d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc30c005610 tx=0x7fc30c005640 comp rx=0 tx=0).stop 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 shutdown_connections 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7fc3000378e0 0x7fc300039da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 --2- 172.21.15.160:0/3862849904 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc3180fede0 0x7fc3180680d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 >> 172.21.15.160:0/3862849904 conn(0x7fc3180fa380 msgr2=0x7fc3180faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 shutdown_connections 2024-06-01T00:47:31.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:31.253+0000 7fc31cad1640 1 -- 172.21.15.160:0/3862849904 wait complete. 2024-06-01T00:47:31.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:31 smithi079 ceph-mon[29356]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-01T00:47:31.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:31 smithi079 ceph-mon[29356]: mgrmap e14: smithi079.emylav(active, since 63s) 2024-06-01T00:47:31.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:31 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/3862849904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:32.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:32.801 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:33.004 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:33.052 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.347+0000 7f5b6cda9640 1 -- 172.21.15.160:0/1937326460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 msgr2=0x7f5b680ff1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.347+0000 7f5b6cda9640 1 --2- 172.21.15.160:0/1937326460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b680ff1f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009880 tx=0x7f5b6002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.347+0000 7f5b6cda9640 1 -- 172.21.15.160:0/1937326460 shutdown_connections 2024-06-01T00:47:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.347+0000 7f5b6cda9640 1 --2- 172.21.15.160:0/1937326460 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b680ff1f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.347+0000 7f5b6cda9640 1 -- 172.21.15.160:0/1937326460 >> 172.21.15.160:0/1937326460 conn(0x7f5b680fa370 msgr2=0x7f5b680fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.347+0000 7f5b6cda9640 1 -- 172.21.15.160:0/1937326460 shutdown_connections 2024-06-01T00:47:34.349 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.347+0000 7f5b6cda9640 1 -- 172.21.15.160:0/1937326460 wait complete. 2024-06-01T00:47:34.350 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.348+0000 7f5b6cda9640 1 Processor -- start 2024-06-01T00:47:34.350 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.348+0000 7f5b6cda9640 1 -- start start 2024-06-01T00:47:34.350 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.348+0000 7f5b6cda9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b68068d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:34.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.348+0000 7f5b6cda9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b68069260 con 0x7f5b680fedd0 2024-06-01T00:47:34.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.349+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b68068d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:34.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.349+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b68068d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:46528/0 (socket says 172.21.15.160:46528) 2024-06-01T00:47:34.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.349+0000 7f5b6659c640 1 -- 172.21.15.160:0/3064005290 learned_addr learned my addr 172.21.15.160:0/3064005290 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:34.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.349+0000 7f5b6659c640 1 -- 172.21.15.160:0/3064005290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60009560 con 0x7f5b680fedd0 2024-06-01T00:47:34.351 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.349+0000 7f5b6659c640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b68068d20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009230 tx=0x7f5b600056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:34.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.350+0000 7f5b5f7fe640 1 -- 172.21.15.160:0/3064005290 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b60004aa0 con 0x7f5b680fedd0 2024-06-01T00:47:34.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.350+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b681981a0 con 0x7f5b680fedd0 2024-06-01T00:47:34.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.350+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b681985e0 con 0x7f5b680fedd0 2024-06-01T00:47:34.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.350+0000 7f5b5f7fe640 1 -- 172.21.15.160:0/3064005290 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b60004c00 con 0x7f5b680fedd0 2024-06-01T00:47:34.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.350+0000 7f5b5f7fe640 1 -- 172.21.15.160:0/3064005290 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b60031a60 con 0x7f5b680fedd0 2024-06-01T00:47:34.352 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.351+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b68061ae0 con 0x7f5b680fedd0 2024-06-01T00:47:34.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.351+0000 7f5b5f7fe640 1 -- 172.21.15.160:0/3064005290 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 14) v1 ==== 44936+0+0 (secure 0 0 0) 0x7f5b60031bc0 con 0x7f5b680fedd0 2024-06-01T00:47:34.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.352+0000 7f5b5f7fe640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 0x7f5b54039da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:34.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.352+0000 7f5b5f7fe640 1 -- 172.21.15.160:0/3064005290 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b6006bb70 con 0x7f5b680fedd0 2024-06-01T00:47:34.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.352+0000 7f5b65d9b640 1 -- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 msgr2=0x7f5b54039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3983389754 2024-06-01T00:47:34.353 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.352+0000 7f5b65d9b640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 0x7f5b54039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-01T00:47:34.357 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.356+0000 7f5b5f7fe640 1 -- 172.21.15.160:0/3064005290 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5b60004d70 con 0x7f5b680fedd0 2024-06-01T00:47:34.554 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.553+0000 7f5b65d9b640 1 -- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 msgr2=0x7f5b54039da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3983389754 2024-06-01T00:47:34.554 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.553+0000 7f5b65d9b640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 0x7f5b54039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-01T00:47:34.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.595+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b6818dbf0 con 0x7f5b680fedd0 2024-06-01T00:47:34.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.596+0000 7f5b5f7fe640 1 -- 172.21.15.160:0/3064005290 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f5b60040300 con 0x7f5b680fedd0 2024-06-01T00:47:34.597 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:34.597 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:34.598 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.597+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 msgr2=0x7f5b54039da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.597+0000 7f5b6cda9640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 0x7f5b54039da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.597+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 msgr2=0x7f5b68068d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.597+0000 7f5b6cda9640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b68068d20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009230 tx=0x7f5b600056e0 comp rx=0 tx=0).stop 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.598+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 shutdown_connections 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.598+0000 7f5b6cda9640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:6800/3983389754,v1:172.21.15.79:6801/3983389754] conn(0x7f5b540378e0 0x7f5b54039da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.598+0000 7f5b6cda9640 1 --2- 172.21.15.160:0/3064005290 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5b680fedd0 0x7f5b68068d20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:34.599 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.598+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 >> 172.21.15.160:0/3064005290 conn(0x7f5b680fa370 msgr2=0x7f5b680faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:34.600 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.598+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 shutdown_connections 2024-06-01T00:47:34.600 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:34.598+0000 7f5b6cda9640 1 -- 172.21.15.160:0/3064005290 wait complete. 2024-06-01T00:47:34.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:34 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/3064005290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:35.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: Active manager daemon smithi079.emylav restarted 2024-06-01T00:47:35.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: Activating manager daemon smithi079.emylav 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: osdmap e5: 0 total, 0 up, 0 in 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: mgrmap e15: smithi079.emylav(active, starting, since 0.0028984s) 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr metadata", "who": "smithi079.emylav", "id": "smithi079.emylav"}]: dispatch 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: Manager daemon smithi079.emylav is now available 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:35.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:47:35.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/mirror_snapshot_schedule"}]: dispatch 2024-06-01T00:47:35.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/trash_purge_schedule"}]: dispatch 2024-06-01T00:47:36.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:36.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:36.384 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:36.439 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-01T00:47:37.182 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:36 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:37.182 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:36 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:37.182 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:36 smithi079 ceph-mon[29356]: mgrmap e16: smithi079.emylav(active, since 1.00591s) 2024-06-01T00:47:37.182 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:36 smithi079 ceph-mon[29356]: [01/Jun/2024:00:47:36] ENGINE Bus STARTING 2024-06-01T00:47:37.182 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:36 smithi079 ceph-mon[29356]: [01/Jun/2024:00:47:36] ENGINE Serving on http://172.21.15.79:8765 2024-06-01T00:47:37.182 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:36 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:37.863 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.860+0000 7fc4590e8640 1 -- 172.21.15.160:0/883182965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 msgr2=0x7fc4540ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:37.863 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.860+0000 7fc4590e8640 1 --2- 172.21.15.160:0/883182965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4540ff200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc444009880 tx=0x7fc44402a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:37.863 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.861+0000 7fc4590e8640 1 -- 172.21.15.160:0/883182965 shutdown_connections 2024-06-01T00:47:37.863 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.861+0000 7fc4590e8640 1 --2- 172.21.15.160:0/883182965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4540ff200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:37.863 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.861+0000 7fc4590e8640 1 -- 172.21.15.160:0/883182965 >> 172.21.15.160:0/883182965 conn(0x7fc4540fa380 msgr2=0x7fc4540fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:37.864 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.861+0000 7fc4590e8640 1 -- 172.21.15.160:0/883182965 shutdown_connections 2024-06-01T00:47:37.864 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.861+0000 7fc4590e8640 1 -- 172.21.15.160:0/883182965 wait complete. 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.862+0000 7fc4590e8640 1 Processor -- start 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.862+0000 7fc4590e8640 1 -- start start 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.862+0000 7fc4590e8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4541975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.862+0000 7fc4590e8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc454197b30 con 0x7fc4540fede0 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc452d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4541975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc452d9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4541975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:46560/0 (socket says 172.21.15.160:46560) 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc452d9d640 1 -- 172.21.15.160:0/627833645 learned_addr learned my addr 172.21.15.160:0/627833645 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc452d9d640 1 -- 172.21.15.160:0/627833645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc444009560 con 0x7fc4540fede0 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc452d9d640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4541975f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc444009230 tx=0x7fc444005610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc444004ae0 con 0x7fc4540fede0 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc444004c40 con 0x7fc4540fede0 2024-06-01T00:47:37.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc444031a80 con 0x7fc4540fede0 2024-06-01T00:47:37.867 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc454197d30 con 0x7fc4540fede0 2024-06-01T00:47:37.867 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.863+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4541981d0 con 0x7fc4540fede0 2024-06-01T00:47:37.867 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.864+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 16) v1 ==== 44787+0+0 (secure 0 0 0) 0x7fc444031cb0 con 0x7fc4540fede0 2024-06-01T00:47:37.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.864+0000 7fc43bfff640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc43c0377b0 0x7fc43c039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:37.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.864+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fc44406c1f0 con 0x7fc4540fede0 2024-06-01T00:47:37.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.865+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4541044a0 con 0x7fc4540fede0 2024-06-01T00:47:37.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.865+0000 7fc45259c640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc43c0377b0 0x7fc43c039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:37.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.865+0000 7fc45259c640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc43c0377b0 0x7fc43c039c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc4480098c0 tx=0x7fc448006db0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:37.871 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.870+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc444035080 con 0x7fc4540fede0 2024-06-01T00:47:37.877 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:37.876+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7fc44403c890 con 0x7fc4540fede0 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:37 smithi079 ceph-mon[29356]: [01/Jun/2024:00:47:36] ENGINE Serving on https://172.21.15.79:7150 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:37 smithi079 ceph-mon[29356]: [01/Jun/2024:00:47:36] ENGINE Bus STARTED 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:37 smithi079 ceph-mon[29356]: [01/Jun/2024:00:47:36] ENGINE Error in HTTPServer.serve 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Traceback (most recent call last): 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-01T00:47:38.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._run(expiration_interval) 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: s = self.context.wrap_socket( 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: return self.sslsocket_class._create( 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self.do_handshake() 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: self._sslobj.do_handshake() 2024-06-01T00:47:38.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-01T00:47:38.118 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.116+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc45402c3e0 con 0x7fc4540fede0 2024-06-01T00:47:38.119 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.117+0000 7fc43bfff640 1 -- 172.21.15.160:0/627833645 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc444036c70 con 0x7fc4540fede0 2024-06-01T00:47:38.119 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:38.119 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:38.119 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:38.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc43c0377b0 msgr2=0x7fc43c039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:38.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc43c0377b0 0x7fc43c039c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc4480098c0 tx=0x7fc448006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:38.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 msgr2=0x7fc4541975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:38.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4541975f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc444009230 tx=0x7fc444005610 comp rx=0 tx=0).stop 2024-06-01T00:47:38.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 shutdown_connections 2024-06-01T00:47:38.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc43c0377b0 0x7fc43c039c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:38.122 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 --2- 172.21.15.160:0/627833645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4540fede0 0x7fc4541975f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:38.122 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 >> 172.21.15.160:0/627833645 conn(0x7fc4540fa380 msgr2=0x7fc4540fae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:38.122 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 shutdown_connections 2024-06-01T00:47:38.122 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:38.119+0000 7fc4590e8640 1 -- 172.21.15.160:0/627833645 wait complete. 2024-06-01T00:47:39.155 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:38 smithi079 ceph-mon[29356]: mgrmap e17: smithi079.emylav(active, since 2s) 2024-06-01T00:47:39.155 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:38 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/627833645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:40.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:40.699 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:40.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:47:40.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:40.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:47:40.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:47:40.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:40 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:47:40.897 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:47:41.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: Updating smithi079:/etc/ceph/ceph.conf 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: Updating smithi160:/etc/ceph/ceph.conf 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-06-01T00:47:41.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:47:42.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:42 smithi079 ceph-mon[29356]: Updating smithi079:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:47:42.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:42 smithi079 ceph-mon[29356]: Updating smithi160:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:47:42.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:42 smithi079 ceph-mon[29356]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-06-01T00:47:42.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:42 smithi079 ceph-mon[29356]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-06-01T00:47:42.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:42 smithi079 ceph-mon[29356]: Updating smithi079:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T00:47:42.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:42 smithi079 ceph-mon[29356]: Updating smithi160:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T00:47:42.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:42 smithi079 ceph-mon[29356]: Deploying daemon ceph-exporter.smithi160 on smithi160 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.904+0000 7ff078e9b640 1 -- 172.21.15.160:0/206732361 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 msgr2=0x7ff0741051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.904+0000 7ff078e9b640 1 --2- 172.21.15.160:0/206732361 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff0741051e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff064009880 tx=0x7ff06402a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.904+0000 7ff078e9b640 1 -- 172.21.15.160:0/206732361 shutdown_connections 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.904+0000 7ff078e9b640 1 --2- 172.21.15.160:0/206732361 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff0741051e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.904+0000 7ff078e9b640 1 -- 172.21.15.160:0/206732361 >> 172.21.15.160:0/206732361 conn(0x7ff0740748c0 msgr2=0x7ff074074cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.904+0000 7ff078e9b640 1 -- 172.21.15.160:0/206732361 shutdown_connections 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.904+0000 7ff078e9b640 1 -- 172.21.15.160:0/206732361 wait complete. 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.905+0000 7ff078e9b640 1 Processor -- start 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.905+0000 7ff078e9b640 1 -- start start 2024-06-01T00:47:43.271 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.905+0000 7ff078e9b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff074195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.905+0000 7ff078e9b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff074195cc0 con 0x7ff074104e00 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff07259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff074195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff07259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff074195780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:60108/0 (socket says 172.21.15.160:60108) 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff07259c640 1 -- 172.21.15.160:0/4192548791 learned_addr learned my addr 172.21.15.160:0/4192548791 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff07259c640 1 -- 172.21.15.160:0/4192548791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff064009560 con 0x7ff074104e00 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff07259c640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff074195780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff064009230 tx=0x7ff0640056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff0637fe640 1 -- 172.21.15.160:0/4192548791 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff064004be0 con 0x7ff074104e00 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff074195f20 con 0x7ff074104e00 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff0637fe640 1 -- 172.21.15.160:0/4192548791 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff064004d40 con 0x7ff074104e00 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff0637fe640 1 -- 172.21.15.160:0/4192548791 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff064031a90 con 0x7ff074104e00 2024-06-01T00:47:43.272 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.906+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0741963a0 con 0x7ff074104e00 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.907+0000 7ff0637fe640 1 -- 172.21.15.160:0/4192548791 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7ff064031cc0 con 0x7ff074104e00 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.907+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0741004b0 con 0x7ff074104e00 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.907+0000 7ff0637fe640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff05c037980 0x7ff05c039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.908+0000 7ff0637fe640 1 -- 172.21.15.160:0/4192548791 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff06406b420 con 0x7ff074104e00 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.908+0000 7ff071d9b640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff05c037980 0x7ff05c039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.908+0000 7ff071d9b640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff05c037980 0x7ff05c039e40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0680098c0 tx=0x7ff068006db0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:42.913+0000 7ff0637fe640 1 -- 172.21.15.160:0/4192548791 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff06404d710 con 0x7ff074104e00 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.159+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0741091b0 con 0x7ff074104e00 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.159+0000 7ff0637fe640 1 -- 172.21.15.160:0/4192548791 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff064035300 con 0x7ff074104e00 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff05c037980 msgr2=0x7ff05c039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff05c037980 0x7ff05c039e40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0680098c0 tx=0x7ff068006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:43.273 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 msgr2=0x7ff074195780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff074195780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff064009230 tx=0x7ff0640056e0 comp rx=0 tx=0).stop 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 shutdown_connections 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff05c037980 0x7ff05c039e40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 --2- 172.21.15.160:0/4192548791 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff074104e00 0x7ff074195780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 >> 172.21.15.160:0/4192548791 conn(0x7ff0740748c0 msgr2=0x7ff07418e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 shutdown_connections 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:43.161+0000 7ff078e9b640 1 -- 172.21.15.160:0/4192548791 wait complete. 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:43.274 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:43.757 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:43 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/4192548791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:45.000 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:45.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:45.736 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: Deploying daemon crash.smithi160 on smithi160 2024-06-01T00:47:46.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.072+0000 7fd15c4ad640 1 -- 172.21.15.160:0/3400490069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 msgr2=0x7fd1541051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.072+0000 7fd15c4ad640 1 --2- 172.21.15.160:0/3400490069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd1541051e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd150009880 tx=0x7fd15002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.072+0000 7fd15c4ad640 1 -- 172.21.15.160:0/3400490069 shutdown_connections 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.072+0000 7fd15c4ad640 1 --2- 172.21.15.160:0/3400490069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd1541051e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.072+0000 7fd15c4ad640 1 -- 172.21.15.160:0/3400490069 >> 172.21.15.160:0/3400490069 conn(0x7fd1540748c0 msgr2=0x7fd154074cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.072+0000 7fd15c4ad640 1 -- 172.21.15.160:0/3400490069 shutdown_connections 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.072+0000 7fd15c4ad640 1 -- 172.21.15.160:0/3400490069 wait complete. 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.073+0000 7fd15c4ad640 1 Processor -- start 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.073+0000 7fd15c4ad640 1 -- start start 2024-06-01T00:47:48.075 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.073+0000 7fd15c4ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd154194760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:48.076 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.073+0000 7fd15c4ad640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd154194ca0 con 0x7fd154104e00 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.073+0000 7fd15a249640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd154194760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.073+0000 7fd15a249640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd154194760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:60134/0 (socket says 172.21.15.160:60134) 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.073+0000 7fd15a249640 1 -- 172.21.15.160:0/2020652844 learned_addr learned my addr 172.21.15.160:0/2020652844 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.074+0000 7fd15a249640 1 -- 172.21.15.160:0/2020652844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150009560 con 0x7fd154104e00 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.074+0000 7fd15a249640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd154194760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd150009230 tx=0x7fd1500056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.074+0000 7fd14b7fe640 1 -- 172.21.15.160:0/2020652844 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd15003e070 con 0x7fd154104e00 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.074+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd154194ea0 con 0x7fd154104e00 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.074+0000 7fd14b7fe640 1 -- 172.21.15.160:0/2020652844 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd150004d90 con 0x7fd154104e00 2024-06-01T00:47:48.077 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.074+0000 7fd14b7fe640 1 -- 172.21.15.160:0/2020652844 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd150031650 con 0x7fd154104e00 2024-06-01T00:47:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.074+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd154195320 con 0x7fd154104e00 2024-06-01T00:47:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.075+0000 7fd14b7fe640 1 -- 172.21.15.160:0/2020652844 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7fd1500317b0 con 0x7fd154104e00 2024-06-01T00:47:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.075+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1541004b0 con 0x7fd154104e00 2024-06-01T00:47:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.075+0000 7fd14b7fe640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd1400379d0 0x7fd140039e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.075+0000 7fd14b7fe640 1 -- 172.21.15.160:0/2020652844 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fd15006c5c0 con 0x7fd154104e00 2024-06-01T00:47:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.076+0000 7fd159a48640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd1400379d0 0x7fd140039e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:48.078 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.076+0000 7fd159a48640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd1400379d0 0x7fd140039e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd1440098c0 tx=0x7fd144006db0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:48.082 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.081+0000 7fd14b7fe640 1 -- 172.21.15.160:0/2020652844 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd15004ab90 con 0x7fd154104e00 2024-06-01T00:47:48.341 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.339+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd154061ae0 con 0x7fd154104e00 2024-06-01T00:47:48.342 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.339+0000 7fd14b7fe640 1 -- 172.21.15.160:0/2020652844 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fd150035020 con 0x7fd154104e00 2024-06-01T00:47:48.342 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:48.342 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:48.342 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd1400379d0 msgr2=0x7fd140039e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd1400379d0 0x7fd140039e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd1440098c0 tx=0x7fd144006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 msgr2=0x7fd154194760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd154194760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd150009230 tx=0x7fd1500056e0 comp rx=0 tx=0).stop 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 shutdown_connections 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd1400379d0 0x7fd140039e90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 --2- 172.21.15.160:0/2020652844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd154104e00 0x7fd154194760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 >> 172.21.15.160:0/2020652844 conn(0x7fd1540748c0 msgr2=0x7fd1540fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 shutdown_connections 2024-06-01T00:47:48.344 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:48.341+0000 7fd15c4ad640 1 -- 172.21.15.160:0/2020652844 wait complete. 2024-06-01T00:47:49.175 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:48 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:49.175 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:48 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:49.175 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:48 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:49.175 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:48 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:49.175 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:48 smithi079 ceph-mon[29356]: Deploying daemon node-exporter.smithi160 on smithi160 2024-06-01T00:47:49.175 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:48 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/2020652844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:49.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:49.956 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:50.170 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:47:50.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:47:50.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:52.222 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.197+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1374214437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 msgr2=0x7f4bb01004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:52.222 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.197+0000 7f4bb7ebf640 1 --2- 172.21.15.160:0/1374214437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb01004a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009880 tx=0x7f4bac02a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.197+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1374214437 shutdown_connections 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.197+0000 7f4bb7ebf640 1 --2- 172.21.15.160:0/1374214437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb01004a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.197+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1374214437 >> 172.21.15.160:0/1374214437 conn(0x7f4bb00fb930 msgr2=0x7f4bb00fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.197+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1374214437 shutdown_connections 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.197+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1374214437 wait complete. 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.198+0000 7f4bb7ebf640 1 Processor -- start 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.198+0000 7f4bb7ebf640 1 -- start start 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.198+0000 7f4bb7ebf640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb019b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.198+0000 7f4bb7ebf640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb019ba60 con 0x7f4bb01000c0 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4bb5c5b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb019b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4bb5c5b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb019b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:33020/0 (socket says 172.21.15.160:33020) 2024-06-01T00:47:52.223 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4bb5c5b640 1 -- 172.21.15.160:0/1934494627 learned_addr learned my addr 172.21.15.160:0/1934494627 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4bb5c5b640 1 -- 172.21.15.160:0/1934494627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac009560 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4bb5c5b640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb019b520 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009230 tx=0x7f4bac0056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4ba6ffd640 1 -- 172.21.15.160:0/1934494627 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bac004aa0 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb019bc60 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4ba6ffd640 1 -- 172.21.15.160:0/1934494627 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bac004c00 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4ba6ffd640 1 -- 172.21.15.160:0/1934494627 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bac031940 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.199+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb019c080 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.200+0000 7f4ba6ffd640 1 -- 172.21.15.160:0/1934494627 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7f4bac0035f0 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.200+0000 7f4ba6ffd640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b9c037980 0x7f4b9c039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.201+0000 7f4ba6ffd640 1 -- 172.21.15.160:0/1934494627 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f4bac06b4f0 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.201+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb01015a0 con 0x7f4bb01000c0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.201+0000 7f4bb545a640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b9c037980 0x7f4b9c039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:52.224 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.201+0000 7f4bb545a640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b9c037980 0x7f4b9c039e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00098c0 tx=0x7f4ba0006db0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:52.225 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.206+0000 7f4ba6ffd640 1 -- 172.21.15.160:0/1934494627 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4bac035080 con 0x7f4bb01000c0 2024-06-01T00:47:52.462 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.461+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bb0061ae0 con 0x7f4bb01000c0 2024-06-01T00:47:52.462 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.461+0000 7f4ba6ffd640 1 -- 172.21.15.160:0/1934494627 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f4bac040020 con 0x7f4bb01000c0 2024-06-01T00:47:52.463 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:52.463 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:52.463 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:52.465 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b9c037980 msgr2=0x7f4b9c039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:52.465 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b9c037980 0x7f4b9c039e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00098c0 tx=0x7f4ba0006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:52.465 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 msgr2=0x7f4bb019b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:52.465 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb019b520 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009230 tx=0x7f4bac0056e0 comp rx=0 tx=0).stop 2024-06-01T00:47:52.466 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 shutdown_connections 2024-06-01T00:47:52.466 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b9c037980 0x7f4b9c039e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:52.466 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 --2- 172.21.15.160:0/1934494627 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4bb01000c0 0x7f4bb019b520 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:52.466 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 >> 172.21.15.160:0/1934494627 conn(0x7f4bb00fb930 msgr2=0x7f4bb00fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:52.466 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 shutdown_connections 2024-06-01T00:47:52.466 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:52.464+0000 7f4bb7ebf640 1 -- 172.21.15.160:0/1934494627 wait complete. 2024-06-01T00:47:52.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:52 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1934494627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:54.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:54.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:54.554 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.dxcuke", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.dxcuke", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: Deploying daemon mgr.smithi160.dxcuke on smithi160 2024-06-01T00:47:55.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:55.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:56.593 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.592+0000 7ff068f17640 1 -- 172.21.15.160:0/352068509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 msgr2=0x7ff0640ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:56.593 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.592+0000 7ff068f17640 1 --2- 172.21.15.160:0/352068509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff0640ff3b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff054009880 tx=0x7ff05402a5d0 comp rx=0 tx=0).stop 2024-06-01T00:47:56.593 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.592+0000 7ff068f17640 1 -- 172.21.15.160:0/352068509 shutdown_connections 2024-06-01T00:47:56.593 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.592+0000 7ff068f17640 1 --2- 172.21.15.160:0/352068509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff0640ff3b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:56.593 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.592+0000 7ff068f17640 1 -- 172.21.15.160:0/352068509 >> 172.21.15.160:0/352068509 conn(0x7ff0640fa8a0 msgr2=0x7ff0640fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:56.594 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.592+0000 7ff068f17640 1 -- 172.21.15.160:0/352068509 shutdown_connections 2024-06-01T00:47:56.594 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.592+0000 7ff068f17640 1 -- 172.21.15.160:0/352068509 wait complete. 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.593+0000 7ff068f17640 1 Processor -- start 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.593+0000 7ff068f17640 1 -- start start 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.593+0000 7ff068f17640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff064194440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.593+0000 7ff068f17640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff064069110 con 0x7ff0640fefd0 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.593+0000 7ff06259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff064194440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.593+0000 7ff06259c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff064194440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:33032/0 (socket says 172.21.15.160:33032) 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.593+0000 7ff06259c640 1 -- 172.21.15.160:0/2909412499 learned_addr learned my addr 172.21.15.160:0/2909412499 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.594+0000 7ff06259c640 1 -- 172.21.15.160:0/2909412499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff054009560 con 0x7ff0640fefd0 2024-06-01T00:47:56.595 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.594+0000 7ff06259c640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff064194440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff054009230 tx=0x7ff0540056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:56.596 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.594+0000 7ff05b7fe640 1 -- 172.21.15.160:0/2909412499 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff054004aa0 con 0x7ff0640fefd0 2024-06-01T00:47:56.596 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.594+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff064194980 con 0x7ff0640fefd0 2024-06-01T00:47:56.596 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.594+0000 7ff05b7fe640 1 -- 172.21.15.160:0/2909412499 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff054004c00 con 0x7ff0640fefd0 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.594+0000 7ff05b7fe640 1 -- 172.21.15.160:0/2909412499 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff054031940 con 0x7ff0640fefd0 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.594+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff064194da0 con 0x7ff0640fefd0 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.595+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0641004b0 con 0x7ff0640fefd0 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.595+0000 7ff05b7fe640 1 -- 172.21.15.160:0/2909412499 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7ff054031b00 con 0x7ff0640fefd0 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.595+0000 7ff05b7fe640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff050037980 0x7ff050039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.595+0000 7ff05b7fe640 1 -- 172.21.15.160:0/2909412499 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff05406b510 con 0x7ff0640fefd0 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.596+0000 7ff05bfff640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff050037980 0x7ff050039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:47:56.597 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.596+0000 7ff05bfff640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff050037980 0x7ff050039e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0098c0 tx=0x7ff04c006db0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:47:56.601 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.600+0000 7ff05b7fe640 1 -- 172.21.15.160:0/2909412499 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff05403a590 con 0x7ff0640fefd0 2024-06-01T00:47:56.843 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.842+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff064068a70 con 0x7ff0640fefd0 2024-06-01T00:47:56.844 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.843+0000 7ff05b7fe640 1 -- 172.21.15.160:0/2909412499 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7ff054040020 con 0x7ff0640fefd0 2024-06-01T00:47:56.844 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:47:56.844 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:45:20.164304Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T00:47:56.845 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-06-01T00:47:56.846 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff050037980 msgr2=0x7ff050039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:56.846 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff050037980 0x7ff050039e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0098c0 tx=0x7ff04c006db0 comp rx=0 tx=0).stop 2024-06-01T00:47:56.846 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 msgr2=0x7ff064194440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:47:56.846 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff064194440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff054009230 tx=0x7ff0540056e0 comp rx=0 tx=0).stop 2024-06-01T00:47:56.847 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 shutdown_connections 2024-06-01T00:47:56.847 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff050037980 0x7ff050039e40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:56.847 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 --2- 172.21.15.160:0/2909412499 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff0640fefd0 0x7ff064194440 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:47:56.847 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 >> 172.21.15.160:0/2909412499 conn(0x7ff0640fa8a0 msgr2=0x7ff0640fb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:47:56.847 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 shutdown_connections 2024-06-01T00:47:56.847 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:47:56.845+0000 7ff068f17640 1 -- 172.21.15.160:0/2909412499 wait complete. 2024-06-01T00:47:57.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:56 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/2909412499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:47:58.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:57 smithi079 ceph-mon[29356]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:47:58.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-01T00:47:58.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T00:47:59.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:59 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:59.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:59 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:59.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:59 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:59.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:59 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:47:59.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:59 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T00:47:59.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:59 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:47:59.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:47:59 smithi079 ceph-mon[29356]: Deploying daemon mon.smithi160 on smithi160 2024-06-01T00:48:00.549 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:48:00.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:00 smithi079 ceph-mon[29356]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:01.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 podman[36594]: 2024-06-01 00:48:02.22154506 +0000 UTC m=+0.122444944 container create 52e1af6da32d65a59cb9693cb668b833294d0eb681e880c85210aa802316192d (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_CLEAN=True, RELEASE=reef-71c5cfc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, ceph=True) 2024-06-01T00:48:02.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:02 smithi079 ceph-mon[29356]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.557+0000 7fb78efb0640 1 -- 172.21.15.160:0/3956754569 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 msgr2=0x7fb7880ff4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.557+0000 7fb78efb0640 1 --2- 172.21.15.160:0/3956754569 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7880ff4e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb784009880 tx=0x7fb78402a5d0 comp rx=0 tx=0).stop 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.557+0000 7fb78efb0640 1 -- 172.21.15.160:0/3956754569 shutdown_connections 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.557+0000 7fb78efb0640 1 --2- 172.21.15.160:0/3956754569 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7880ff4e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.557+0000 7fb78efb0640 1 -- 172.21.15.160:0/3956754569 >> 172.21.15.160:0/3956754569 conn(0x7fb788075b60 msgr2=0x7fb7880fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.557+0000 7fb78efb0640 1 -- 172.21.15.160:0/3956754569 shutdown_connections 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.557+0000 7fb78efb0640 1 -- 172.21.15.160:0/3956754569 wait complete. 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.558+0000 7fb78efb0640 1 Processor -- start 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.558+0000 7fb78efb0640 1 -- start start 2024-06-01T00:48:02.793 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.558+0000 7fb78efb0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7881967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.558+0000 7fb78efb0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788068b20 con 0x7fb7880ff100 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.558+0000 7fb78cd4c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7881967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.558+0000 7fb78cd4c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7881967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.160:50726/0 (socket says 172.21.15.160:50726) 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.558+0000 7fb78cd4c640 1 -- 172.21.15.160:0/1956512767 learned_addr learned my addr 172.21.15.160:0/1956512767 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.559+0000 7fb78cd4c640 1 -- 172.21.15.160:0/1956512767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784009560 con 0x7fb7880ff100 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.559+0000 7fb78cd4c640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7881967b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb784009230 tx=0x7fb7840056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.559+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb784004aa0 con 0x7fb7880ff100 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.559+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb784004c00 con 0x7fb7880ff100 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.559+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb788196d50 con 0x7fb7880ff100 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.559+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb784031940 con 0x7fb7880ff100 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.559+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7881971d0 con 0x7fb7880ff100 2024-06-01T00:48:02.794 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.560+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 44979+0+0 (secure 0 0 0) 0x7fb784031b00 con 0x7fb7880ff100 2024-06-01T00:48:02.795 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.560+0000 7fb77dffb640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb770037980 0x7fb770039e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:02.795 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.560+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fb78406b4e0 con 0x7fb7880ff100 2024-06-01T00:48:02.795 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.560+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7881005e0 con 0x7fb7880ff100 2024-06-01T00:48:02.795 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.561+0000 7fb77ffff640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb770037980 0x7fb770039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:02.795 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.561+0000 7fb77ffff640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb770037980 0x7fb770039e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb7780098c0 tx=0x7fb778006db0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:02.795 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.566+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb784040080 con 0x7fb7880ff100 2024-06-01T00:48:02.795 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.640+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78403c470 con 0x7fb7880ff100 2024-06-01T00:48:02.831 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:02.830+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb788197450 con 0x7fb7880ff100 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 podman[36594]: 2024-06-01 00:48:02.576555653 +0000 UTC m=+0.477455547 container init 52e1af6da32d65a59cb9693cb668b833294d0eb681e880c85210aa802316192d (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160, org.label-schema.vendor=CentOS, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, RELEASE=reef-71c5cfc) 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 podman[36594]: 2024-06-01 00:48:02.579869816 +0000 UTC m=+0.480769707 container start 52e1af6da32d65a59cb9693cb668b833294d0eb681e880c85210aa802316192d (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=reef-71c5cfc, org.label-schema.vendor=CentOS, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: set uid:gid to 167:167 (ceph:ceph) 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: ceph version 18.2.2-1387-g71c5cfc9 (71c5cfc99f93c162032256d34d9b60b8288fad91) reef (stable), process ceph-mon, pid 2 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pidfile_write: ignore empty --pid-file 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: load: jerasure load: lrc 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: RocksDB version: 7.9.2 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Git sha 0 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Compile date 2024-05-31 16:05:45 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: DB SUMMARY 2024-06-01T00:48:02.832 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: DB Session ID: X9449GAPNT7H6HHLCNKA 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: CURRENT file: CURRENT 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: IDENTITY file: IDENTITY 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi160/store.db dir, Total Num: 0, files: 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi160/store.db: 000004.log size: 511 ; 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.error_if_exists: 0 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.create_if_missing: 0 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.paranoid_checks: 1 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-01T00:48:02.833 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.env: 0x5589efe4eba0 2024-06-01T00:48:02.834 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.fs: PosixFileSystem 2024-06-01T00:48:02.834 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.info_log: 0x5589f1080dc0 2024-06-01T00:48:02.834 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_file_opening_threads: 16 2024-06-01T00:48:02.834 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.statistics: (nil) 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.use_fsync: 0 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_log_file_size: 0 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.keep_log_file_num: 1000 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.recycle_log_file_num: 0 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.allow_fallocate: 1 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.allow_mmap_reads: 0 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.allow_mmap_writes: 0 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.use_direct_reads: 0 2024-06-01T00:48:02.835 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.create_missing_column_families: 0 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.db_log_dir: 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.wal_dir: 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.advise_random_on_open: 1 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.db_write_buffer_size: 0 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.write_buffer_manager: 0x5589f1052140 2024-06-01T00:48:02.836 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.rate_limiter: (nil) 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.wal_recovery_mode: 2 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.enable_thread_tracking: 0 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.enable_pipelined_write: 0 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.unordered_write: 0 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-01T00:48:02.837 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.row_cache: None 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.wal_filter: None 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.allow_ingest_behind: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.two_write_queues: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.manual_wal_flush: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.wal_compression: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.atomic_flush: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.log_readahead_size: 0 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-01T00:48:02.838 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.best_efforts_recovery: 0 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.allow_data_in_errors: 0 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.db_host_id: __hostname__ 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_background_jobs: 2 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_background_compactions: -1 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_subcompactions: 1 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_total_wal_size: 0 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-01T00:48:02.839 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_open_files: -1 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bytes_per_sync: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_readahead_size: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_background_flushes: -1 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Compression algorithms supported: 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kZSTD supported: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kXpressCompression supported: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kBZip2Compression supported: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-01T00:48:02.840 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kLZ4Compression supported: 1 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kZlibCompression supported: 1 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kLZ4HCCompression supported: 1 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: kSnappyCompression supported: 1 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi160/store.db/MANIFEST-000005 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.merge_operator: 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_filter: None 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_filter_factory: None 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.sst_partitioner_factory: None 2024-06-01T00:48:02.841 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5589f1080a60) 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: cache_index_and_filter_blocks: 1 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: pin_top_level_index_and_filter: 1 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: index_type: 0 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: data_block_index_type: 0 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: index_shortening: 1 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: checksum: 4 2024-06-01T00:48:02.842 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: no_block_cache: 0 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_cache: 0x5589f1077350 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_cache_name: BinnedLRUCache 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_cache_options: 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: capacity : 536870912 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: num_shard_bits : 4 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: strict_capacity_limit : 0 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: high_pri_pool_ratio: 0.000 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_cache_compressed: (nil) 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: persistent_cache: (nil) 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_size: 4096 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_size_deviation: 10 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_restart_interval: 16 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: index_block_restart_interval: 1 2024-06-01T00:48:02.843 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: metadata_block_size: 4096 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: partition_filters: 0 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: use_delta_encoding: 1 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: filter_policy: bloomfilter 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: whole_key_filtering: 1 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: verify_compression: 0 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: read_amp_bytes_per_bit: 0 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: format_version: 5 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: enable_index_compression: 1 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: block_align: 0 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: max_auto_readahead_size: 262144 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: prepopulate_block_cache: 0 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: initial_auto_readahead_size: 8192 2024-06-01T00:48:02.844 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: num_file_reads_for_auto_readahead: 2 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.write_buffer_size: 33554432 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_write_buffer_number: 2 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression: NoCompression 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression: Disabled 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.prefix_extractor: nullptr 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.num_levels: 7 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-01T00:48:02.845 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.level: 32767 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.strategy: 0 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-01T00:48:02.846 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.enabled: false 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.target_file_size_base: 67108864 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-01T00:48:02.847 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-01T00:48:02.849 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.arena_block_size: 1048576 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.disable_auto_compactions: 0 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-01T00:48:02.850 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.table_properties_collectors: 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.inplace_update_support: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.bloom_locality: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.max_successive_merges: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.paranoid_file_checks: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.force_consistency_checks: 1 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.report_bg_io_stats: 0 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.ttl: 2592000 2024-06-01T00:48:02.851 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.enable_blob_files: false 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.min_blob_size: 0 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.blob_file_size: 268435456 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.blob_file_starting_level: 0 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi160/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-01T00:48:02.852 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8abf09cd-55bf-4151-add2-a064978850c4 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717202882606913, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717202882607363, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717202882, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8abf09cd-55bf-4151-add2-a064978850c4", "db_session_id": "X9449GAPNT7H6HHLCNKA", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717202882607477, "job": 1, "event": "recovery_finished"} 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi160/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5589f10a0e00 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: DB pointer 0x5589f1194000 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: ** DB Stats ** 2024-06-01T00:48:02.853 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: ** Compaction Stats [default] ** 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-01T00:48:02.854 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: ** Compaction Stats [default] ** 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T00:48:02.855 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Block cache BinnedLRUCache@0x5589f1077350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160 does not exist in monmap, will attempt to join an existing cluster 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: using public_addr v2:172.21.15.160:0/0 -> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: starting mon.smithi160 rank -1 at public addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] at bind addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi160 fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(???) e0 preinit fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).mds e1 new map 2024-06-01T00:48:02.856 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).mds e1 print_map 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: e1 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: legacy client fscid: -1 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: No filesystems configured 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-01T00:48:02.857 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.858 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon node-exporter.smithi079 on smithi079 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.14191 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying cephadm binary to smithi160 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Added host smithi160 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon alertmanager.smithi079 on smithi079 2024-06-01T00:48:02.859 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1979565680' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1979565680' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Saving service mon spec with placement smithi079:172.21.15.79=smithi079;smithi160:172.21.15.160=smithi160;count:2 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.860 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon grafana.smithi079 on smithi079 2024-06-01T00:48:02.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/4027875508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1707632200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1002713913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1557245373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.862 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/550060646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon prometheus.smithi079 on smithi079 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1395763300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.863 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/3837092344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/2586726844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14162 172.21.15.79:0/3006235589' entity='mgr.smithi079.emylav' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mgrmap e14: smithi079.emylav(active, since 63s) 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/3862849904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.864 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/3064005290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Active manager daemon smithi079.emylav restarted 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Activating manager daemon smithi079.emylav 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: osdmap e5: 0 total, 0 up, 0 in 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mgrmap e15: smithi079.emylav(active, starting, since 0.0028984s) 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr metadata", "who": "smithi079.emylav", "id": "smithi079.emylav"}]: dispatch 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Manager daemon smithi079.emylav is now available 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.865 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/mirror_snapshot_schedule"}]: dispatch 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.emylav/trash_purge_schedule"}]: dispatch 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mgrmap e16: smithi079.emylav(active, since 1.00591s) 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: [01/Jun/2024:00:47:36] ENGINE Bus STARTING 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: [01/Jun/2024:00:47:36] ENGINE Serving on http://172.21.15.79:8765 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: [01/Jun/2024:00:47:36] ENGINE Serving on https://172.21.15.79:7150 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: [01/Jun/2024:00:47:36] ENGINE Bus STARTED 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: [01/Jun/2024:00:47:36] ENGINE Error in HTTPServer.serve 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: Traceback (most recent call last): 2024-06-01T00:48:02.866 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self._connections.run(self.expiration_interval) 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self._run(expiration_interval) 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: s = self.context.wrap_socket( 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: return self.sslsocket_class._create( 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self.do_handshake() 2024-06-01T00:48:02.867 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: self._sslobj.do_handshake() 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mgrmap e17: smithi079.emylav(active, since 2s) 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/627833645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.868 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi079:/etc/ceph/ceph.conf 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi160:/etc/ceph/ceph.conf 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:02.869 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi079:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi160:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi079:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Updating smithi160:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon ceph-exporter.smithi160 on smithi160 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/4192548791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-01T00:48:02.870 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon crash.smithi160 on smithi160 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon node-exporter.smithi160 on smithi160 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/2020652844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1934494627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.871 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.dxcuke", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.dxcuke", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon mgr.smithi160.dxcuke on smithi160 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/2909412499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.872 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: Deploying daemon mon.smithi160 on smithi160 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 ceph-mon[36639]: mon.smithi160@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 bash[36594]: 52e1af6da32d65a59cb9693cb668b833294d0eb681e880c85210aa802316192d 2024-06-01T00:48:02.873 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:02 smithi160 systemd[1]: Started Ceph mon.smithi160 for 22870f5e-1fb0-11ef-bc9b-c7b262605968. 2024-06-01T00:48:07.651 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.650+0000 7fb77dffb640 1 -- 172.21.15.160:0/1956512767 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7fb78404d350 con 0x7fb7880ff100 2024-06-01T00:48:07.652 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-06-01T00:48:07.652 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":2,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","modified":"2024-06-01T00:48:02.639857Z","created":"2024-06-01T00:45:20.164304Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi160","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-01T00:48:07.652 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 2 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb770037980 msgr2=0x7fb770039e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb770037980 0x7fb770039e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb7780098c0 tx=0x7fb778006db0 comp rx=0 tx=0).stop 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 msgr2=0x7fb7881967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7881967b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb784009230 tx=0x7fb7840056e0 comp rx=0 tx=0).stop 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 shutdown_connections 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb770037980 0x7fb770039e40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 --2- 172.21.15.160:0/1956512767 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb7880ff100 0x7fb7881967b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 >> 172.21.15.160:0/1956512767 conn(0x7fb788075b60 msgr2=0x7fb7881069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:48:07.655 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 shutdown_connections 2024-06-01T00:48:07.656 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:48:07.654+0000 7fb78efb0640 1 -- 172.21.15.160:0/1956512767 wait complete. 2024-06-01T00:48:08.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: mon.smithi079 calling monitor election 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1956512767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: Failed to load ceph-mgr modules: k8sevents 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.dxcuke/crt"}]: dispatch 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: mon.smithi160 calling monitor election 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:08.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: mon.smithi079 is new leader, mons smithi079,smithi160 in quorum (ranks 0,1) 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: monmap e2: 2 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],smithi160=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: fsmap 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: osdmap e5: 0 total, 0 up, 0 in 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: mgrmap e17: smithi079.emylav(active, since 32s) 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: Health detail: HEALTH_WARN 1 mgr modules have recently crashed 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: mgr module k8sevents crashed in daemon mgr.smithi079.emylav on host smithi079 at 2024-06-01T00:45:42.153379Z 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: Standby manager daemon smithi160.dxcuke started 2024-06-01T00:48:08.006 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T00:48:08.007 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.007 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.dxcuke/key"}]: dispatch 2024-06-01T00:48:08.007 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T00:48:08.007 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.007 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.007 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.007 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: mon.smithi079 calling monitor election 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1956512767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: Failed to load ceph-mgr modules: k8sevents 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.dxcuke/crt"}]: dispatch 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: mon.smithi160 calling monitor election 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:08.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: mon.smithi079 is new leader, mons smithi079,smithi160 in quorum (ranks 0,1) 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: monmap e2: 2 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],smithi160=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: fsmap 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: osdmap e5: 0 total, 0 up, 0 in 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: mgrmap e17: smithi079.emylav(active, since 32s) 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: Health detail: HEALTH_WARN 1 mgr modules have recently crashed 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: mgr module k8sevents crashed in daemon mgr.smithi079.emylav on host smithi079 at 2024-06-01T00:45:42.153379Z 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: Standby manager daemon smithi160.dxcuke started 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T00:48:08.108 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.109 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi160.dxcuke/key"}]: dispatch 2024-06-01T00:48:08.109 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.? 172.21.15.160:0/1475938798' entity='mgr.smithi160.dxcuke' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T00:48:08.109 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.109 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.109 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.109 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:08.276 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-01T00:48:08.276 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-01T00:48:08.490 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:48:08.943 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:08 smithi079 ceph-mon[29356]: mgrmap e18: smithi079.emylav(active, since 32s), standbys: smithi160.dxcuke 2024-06-01T00:48:08.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:08 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr metadata", "who": "smithi160.dxcuke", "id": "smithi160.dxcuke"}]: dispatch 2024-06-01T00:48:08.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:08 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:48:08.944 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:08 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:09.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:08 smithi160 ceph-mon[36639]: mgrmap e18: smithi079.emylav(active, since 32s), standbys: smithi160.dxcuke 2024-06-01T00:48:09.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:08 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr metadata", "who": "smithi160.dxcuke", "id": "smithi160.dxcuke"}]: dispatch 2024-06-01T00:48:09.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:08 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:48:09.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:08 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:48:09.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:09 smithi079 ceph-mon[29356]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:10.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:09 smithi160 ceph-mon[36639]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:10.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.092+0000 7f5eb4887640 1 -- 172.21.15.79:0/996495103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 msgr2=0x7f5eb01051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:10.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.092+0000 7f5eb4887640 1 --2- 172.21.15.79:0/996495103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb01051e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009880 tx=0x7f5ea002a5d0 comp rx=0 tx=0).stop 2024-06-01T00:48:10.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.093+0000 7f5eb4887640 1 -- 172.21.15.79:0/996495103 shutdown_connections 2024-06-01T00:48:10.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.093+0000 7f5eb4887640 1 --2- 172.21.15.79:0/996495103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb01051e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:10.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.093+0000 7f5eb4887640 1 -- 172.21.15.79:0/996495103 >> 172.21.15.79:0/996495103 conn(0x7f5eb00748c0 msgr2=0x7f5eb0074cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.093+0000 7f5eb4887640 1 -- 172.21.15.79:0/996495103 shutdown_connections 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.093+0000 7f5eb4887640 1 -- 172.21.15.79:0/996495103 wait complete. 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.093+0000 7f5eb4887640 1 Processor -- start 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.093+0000 7f5eb4887640 1 -- start start 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eb4887640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eb4887640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5eb019ab20 0x7f5eb0194720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eb4887640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb019b0f0 con 0x7f5eb0104e00 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eb4887640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb0194c60 con 0x7f5eb019ab20 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eae59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eae59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb019a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36032/0 (socket says 172.21.15.79:36032) 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eae59c640 1 -- 172.21.15.79:0/3987613384 learned_addr learned my addr 172.21.15.79:0/3987613384 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5ea7fff640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5eb019ab20 0x7f5eb0194720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:10.146 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eae59c640 1 -- 172.21.15.79:0/3987613384 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5eb019ab20 msgr2=0x7f5eb0194720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eae59c640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5eb019ab20 0x7f5eb0194720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eae59c640 1 -- 172.21.15.79:0/3987613384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea0009560 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eae59c640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb019a5e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ea00055a0 tx=0x7f5ea00055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5ea77fe640 1 -- 172.21.15.79:0/3987613384 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea0004aa0 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb0194f40 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5ea77fe640 1 -- 172.21.15.79:0/3987613384 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ea0004c00 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5ea77fe640 1 -- 172.21.15.79:0/3987613384 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ea0003a30 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.094+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb0195490 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.095+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb01004b0 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.096+0000 7f5ea77fe640 1 -- 172.21.15.79:0/3987613384 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7f5ea0003b90 con 0x7f5eb0104e00 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.096+0000 7f5ea77fe640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f5e9c06af70 0x7f5e9c06d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:10.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.097+0000 7f5ea77fe640 1 -- 172.21.15.79:0/3987613384 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f5ea00a9f40 con 0x7f5eb0104e00 2024-06-01T00:48:10.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.097+0000 7f5ea7fff640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f5e9c06af70 0x7f5e9c06d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:10.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.097+0000 7f5ea7fff640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f5e9c06af70 0x7f5e9c06d430 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009950 tx=0x7f5e98008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:10.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.101+0000 7f5ea77fe640 1 -- 172.21.15.79:0/3987613384 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f5ea00762c0 con 0x7f5eb0104e00 2024-06-01T00:48:10.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.336+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5eb0061ae0 con 0x7f5eb0104e00 2024-06-01T00:48:10.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.336+0000 7f5ea77fe640 1 -- 172.21.15.79:0/3987613384 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f5ea00a9700 con 0x7f5eb0104e00 2024-06-01T00:48:10.338 INFO:teuthology.orchestra.run.smithi079.stdout:# minimal ceph.conf for 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:48:10.338 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-06-01T00:48:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:48:10.338 INFO:teuthology.orchestra.run.smithi079.stdout: mon_host = [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.338+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f5e9c06af70 msgr2=0x7f5e9c06d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.338+0000 7f5eb4887640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f5e9c06af70 0x7f5e9c06d430 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009950 tx=0x7f5e98008040 comp rx=0 tx=0).stop 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.338+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 msgr2=0x7f5eb019a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.338+0000 7f5eb4887640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb019a5e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5ea00055a0 tx=0x7f5ea00055d0 comp rx=0 tx=0).stop 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.338+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 shutdown_connections 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.339+0000 7f5eb4887640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f5e9c06af70 0x7f5e9c06d430 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.339+0000 7f5eb4887640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f5eb019ab20 0x7f5eb0194720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.339+0000 7f5eb4887640 1 --2- 172.21.15.79:0/3987613384 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5eb0104e00 0x7f5eb019a5e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:10.342 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.339+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 >> 172.21.15.79:0/3987613384 conn(0x7f5eb00748c0 msgr2=0x7f5eb00fc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:48:10.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.339+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 shutdown_connections 2024-06-01T00:48:10.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:10.339+0000 7f5eb4887640 1 -- 172.21.15.79:0/3987613384 wait complete. 2024-06-01T00:48:10.968 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.968 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.968 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:10.968 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: Updating smithi079:/etc/ceph/ceph.conf 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: Updating smithi160:/etc/ceph/ceph.conf 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: Updating smithi160:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: Updating smithi079:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3987613384' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: Reconfiguring mon.smithi079 (unknown last config time)... 2024-06-01T00:48:10.969 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T00:48:10.970 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T00:48:10.970 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:10.970 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:10 smithi079 ceph-mon[29356]: Reconfiguring daemon mon.smithi079 on smithi079 2024-06-01T00:48:10.989 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-01T00:48:10.989 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:48:10.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-06-01T00:48:11.017 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:48:11.017 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-01T00:48:11.082 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:48:11.082 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.conf 2024-06-01T00:48:11.111 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:48:11.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-01T00:48:11.174 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.174 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.174 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:11.174 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: Updating smithi079:/etc/ceph/ceph.conf 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: Updating smithi160:/etc/ceph/ceph.conf 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: Updating smithi160:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: Updating smithi079:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3987613384' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: Reconfiguring mon.smithi079 (unknown last config time)... 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:11.175 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:10 smithi160 ceph-mon[36639]: Reconfiguring daemon mon.smithi079 on smithi079 2024-06-01T00:48:11.177 INFO:tasks.cephadm:Deploying OSDs... 2024-06-01T00:48:11.177 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:48:11.178 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-06-01T00:48:11.193 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-01T00:48:11.193 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-06-01T00:48:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-01T00:48:11.248 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:11.248 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-01T00:48:11.248 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:11.248 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:11.249 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-06-01 00:47:28.674396241 +0000 2024-06-01T00:48:11.249 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-06-01 00:41:17.444386464 +0000 2024-06-01T00:48:11.249 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-06-01 00:41:17.444386464 +0000 2024-06-01T00:48:11.249 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-06-01 00:41:17.444386464 +0000 2024-06-01T00:48:11.249 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-01T00:48:11.312 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-06-01T00:48:11.312 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-06-01T00:48:11.312 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000191784 s, 2.7 MB/s 2024-06-01T00:48:11.313 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-01T00:48:11.370 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-06-01T00:48:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-01T00:48:11.425 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:11.425 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-01T00:48:11.425 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:11.425 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:11.425 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-06-01 00:47:28.675396218 +0000 2024-06-01T00:48:11.426 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-06-01 00:41:17.767379080 +0000 2024-06-01T00:48:11.426 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-06-01 00:41:17.767379080 +0000 2024-06-01T00:48:11.426 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-06-01 00:41:17.767379080 +0000 2024-06-01T00:48:11.426 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-01T00:48:11.490 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-06-01T00:48:11.490 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-06-01T00:48:11.490 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000178977 s, 2.9 MB/s 2024-06-01T00:48:11.491 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-01T00:48:11.548 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-06-01 00:47:28.675396218 +0000 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-06-01 00:41:18.099371489 +0000 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-06-01 00:41:18.099371489 +0000 2024-06-01T00:48:11.603 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-06-01 00:41:18.099371489 +0000 2024-06-01T00:48:11.604 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-01T00:48:11.666 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-06-01T00:48:11.666 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-06-01T00:48:11.666 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000192311 s, 2.7 MB/s 2024-06-01T00:48:11.667 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-01T00:48:11.723 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-06-01T00:48:11.778 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-01T00:48:11.778 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:11.778 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-01T00:48:11.778 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:11.778 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:11.779 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-06-01 00:47:28.675396218 +0000 2024-06-01T00:48:11.779 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-06-01 00:41:18.433363853 +0000 2024-06-01T00:48:11.779 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-06-01 00:41:18.433363853 +0000 2024-06-01T00:48:11.779 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-06-01 00:41:18.433363853 +0000 2024-06-01T00:48:11.779 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-01T00:48:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-06-01T00:48:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-06-01T00:48:11.841 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000213252 s, 2.4 MB/s 2024-06-01T00:48:11.842 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-01T00:48:11.898 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:48:11.898 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-06-01T00:48:11.918 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-01T00:48:11.918 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-06-01 00:48:00.879596096 +0000 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-06-01 00:41:20.224423966 +0000 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-06-01 00:41:20.224423966 +0000 2024-06-01T00:48:11.976 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-06-01 00:41:20.224423966 +0000 2024-06-01T00:48:11.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-01T00:48:12.043 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-06-01T00:48:12.043 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-06-01T00:48:12.043 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00035495 s, 1.4 MB/s 2024-06-01T00:48:12.044 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-01T00:48:12.104 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-06-01T00:48:12.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:11 smithi079 ceph-mon[29356]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-06-01 00:48:00.879596096 +0000 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-06-01 00:41:20.547418917 +0000 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-06-01 00:41:20.547418917 +0000 2024-06-01T00:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-06-01 00:41:20.547418917 +0000 2024-06-01T00:48:12.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-01T00:48:12.225 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:11 smithi160 ceph-mon[36639]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:12.228 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-06-01T00:48:12.228 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-06-01T00:48:12.228 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000220978 s, 2.3 MB/s 2024-06-01T00:48:12.229 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-01T00:48:12.287 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-06-01T00:48:12.344 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-01T00:48:12.344 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:12.344 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-01T00:48:12.344 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:12.345 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:12.345 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-06-01 00:48:00.880596081 +0000 2024-06-01T00:48:12.345 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-06-01 00:41:20.880413713 +0000 2024-06-01T00:48:12.345 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-06-01 00:41:20.880413713 +0000 2024-06-01T00:48:12.345 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-06-01 00:41:20.880413713 +0000 2024-06-01T00:48:12.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-01T00:48:12.409 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-06-01T00:48:12.409 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-06-01T00:48:12.410 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000214494 s, 2.4 MB/s 2024-06-01T00:48:12.411 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-01T00:48:12.468 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-06-01 00:48:00.880596081 +0000 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-06-01 00:41:21.224408336 +0000 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-06-01 00:41:21.224408336 +0000 2024-06-01T00:48:12.524 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-06-01 00:41:21.224408336 +0000 2024-06-01T00:48:12.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-01T00:48:12.588 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-06-01T00:48:12.588 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-06-01T00:48:12.588 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00019001 s, 2.7 MB/s 2024-06-01T00:48:12.589 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-01T00:48:12.646 INFO:tasks.cephadm:Deploying osd.0 on smithi079 with /dev/vg_nvme/lv_4... 2024-06-01T00:48:12.646 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-01T00:48:13.451 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:13 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:13.451 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:13 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:13.451 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:13 smithi079 ceph-mon[29356]: Reconfiguring mgr.smithi079.emylav (unknown last config time)... 2024-06-01T00:48:13.451 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:13 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi079.emylav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T00:48:13.451 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:13 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T00:48:13.451 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:13 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:13.452 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:13 smithi079 ceph-mon[29356]: Reconfiguring daemon mgr.smithi079.emylav on smithi079 2024-06-01T00:48:13.524 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:48:13.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:13 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:13.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:13 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:13.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:13 smithi160 ceph-mon[36639]: Reconfiguring mgr.smithi079.emylav (unknown last config time)... 2024-06-01T00:48:13.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:13 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi079.emylav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T00:48:13.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:13 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T00:48:13.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:13 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:13.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:13 smithi160 ceph-mon[36639]: Reconfiguring daemon mgr.smithi079.emylav on smithi079 2024-06-01T00:48:14.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:14 smithi160 ceph-mon[36639]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:14.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:14 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:14.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:14 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:14.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:14 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-01T00:48:14.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:14 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:14.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:14 smithi079 ceph-mon[29356]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:14.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:14 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:14.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:14 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:14.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:14 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi079", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-01T00:48:14.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:14 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:15.544 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:15 smithi079 ceph-mon[29356]: Reconfiguring ceph-exporter.smithi079 (monmap changed)... 2024-06-01T00:48:15.545 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:15 smithi079 ceph-mon[29356]: Reconfiguring daemon ceph-exporter.smithi079 on smithi079 2024-06-01T00:48:15.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:15 smithi160 ceph-mon[36639]: Reconfiguring ceph-exporter.smithi079 (monmap changed)... 2024-06-01T00:48:15.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:15 smithi160 ceph-mon[36639]: Reconfiguring daemon ceph-exporter.smithi079 on smithi079 2024-06-01T00:48:15.964 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-01T00:48:16.048 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-01T00:48:16.061 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-06-01T00:48:16.062 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-06-01T00:48:16.062 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112918 s, 929 MB/s 2024-06-01T00:48:16.062 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-06-01T00:48:16.432 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:16 smithi079 ceph-mon[29356]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:16.432 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:16.432 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:16.432 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-01T00:48:16.432 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:16.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:16 smithi160 ceph-mon[36639]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:16.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:16.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:16.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-01T00:48:16.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:16.779 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2024-06-01T00:48:17.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:17 smithi160 ceph-mon[36639]: Reconfiguring crash.smithi079 (monmap changed)... 2024-06-01T00:48:17.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:17 smithi160 ceph-mon[36639]: Reconfiguring daemon crash.smithi079 on smithi079 2024-06-01T00:48:17.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:17 smithi079 ceph-mon[29356]: Reconfiguring crash.smithi079 (monmap changed)... 2024-06-01T00:48:17.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:17 smithi079 ceph-mon[29356]: Reconfiguring daemon crash.smithi079 on smithi079 2024-06-01T00:48:17.977 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:48:18.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:18 smithi160 ceph-mon[36639]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:18.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:18 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:18.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:18 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:18.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:18 smithi079 ceph-mon[29356]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:18.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:18 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:18.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:18 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:19.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:19 smithi079 ceph-mon[29356]: Reconfiguring alertmanager.smithi079 (dependencies changed)... 2024-06-01T00:48:19.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:19 smithi079 ceph-mon[29356]: Reconfiguring daemon alertmanager.smithi079 on smithi079 2024-06-01T00:48:19.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:19 smithi160 ceph-mon[36639]: Reconfiguring alertmanager.smithi079 (dependencies changed)... 2024-06-01T00:48:19.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:19 smithi160 ceph-mon[36639]: Reconfiguring daemon alertmanager.smithi079 on smithi079 2024-06-01T00:48:20.542 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:20 smithi079 ceph-mon[29356]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:20.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:20 smithi160 ceph-mon[36639]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:21.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 -- 172.21.15.79:0/2845674973 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30100290 msgr2=0x7fde301006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:21.111 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 --2- 172.21.15.79:0/2845674973 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30100290 0x7fde301006b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fde20009880 tx=0x7fde2002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 -- 172.21.15.79:0/2845674973 shutdown_connections 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 --2- 172.21.15.79:0/2845674973 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde30101490 0x7fde30101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 --2- 172.21.15.79:0/2845674973 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30100290 0x7fde301006b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 -- 172.21.15.79:0/2845674973 >> 172.21.15.79:0/2845674973 conn(0x7fde300fb850 msgr2=0x7fde300fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 -- 172.21.15.79:0/2845674973 shutdown_connections 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 -- 172.21.15.79:0/2845674973 wait complete. 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 Processor -- start 2024-06-01T00:48:21.112 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.110+0000 7fde36853640 1 -- start start 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde36853640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde30100290 0x7fde30074200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde36853640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30101490 0x7fde300728a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde36853640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde30072de0 con 0x7fde30101490 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde36853640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde30072f50 con 0x7fde30100290 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30101490 0x7fde300728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2f7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30101490 0x7fde300728a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49464/0 (socket says 172.21.15.79:49464) 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2f7fe640 1 -- 172.21.15.79:0/553047844 learned_addr learned my addr 172.21.15.79:0/553047844 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2ffff640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde30100290 0x7fde30074200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2f7fe640 1 -- 172.21.15.79:0/553047844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde30100290 msgr2=0x7fde30074200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2f7fe640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde30100290 0x7fde30074200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:21.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2f7fe640 1 -- 172.21.15.79:0/553047844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde20009560 con 0x7fde30101490 2024-06-01T00:48:21.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2f7fe640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30101490 0x7fde300728a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fde2400b420 tx=0x7fde2400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:21.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2d7fa640 1 -- 172.21.15.79:0/553047844 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde24011800 con 0x7fde30101490 2024-06-01T00:48:21.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde30073230 con 0x7fde30101490 2024-06-01T00:48:21.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2d7fa640 1 -- 172.21.15.79:0/553047844 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde24011e20 con 0x7fde30101490 2024-06-01T00:48:21.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.111+0000 7fde2d7fa640 1 -- 172.21.15.79:0/553047844 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde2400f2e0 con 0x7fde30101490 2024-06-01T00:48:21.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.112+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3019dea0 con 0x7fde30101490 2024-06-01T00:48:21.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.112+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3018fe80 con 0x7fde30101490 2024-06-01T00:48:21.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.113+0000 7fde2d7fa640 1 -- 172.21.15.79:0/553047844 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7fde24010380 con 0x7fde30101490 2024-06-01T00:48:21.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.113+0000 7fde2d7fa640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde1806ae70 0x7fde1806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:21.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.114+0000 7fde2ffff640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde1806ae70 0x7fde1806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:21.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.114+0000 7fde2d7fa640 1 -- 172.21.15.79:0/553047844 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fde240887a0 con 0x7fde30101490 2024-06-01T00:48:21.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.114+0000 7fde2ffff640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde1806ae70 0x7fde1806d330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde20009230 tx=0x7fde20005b50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:21.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.118+0000 7fde2d7fa640 1 -- 172.21.15.79:0/553047844 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fde24054b20 con 0x7fde30101490 2024-06-01T00:48:21.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:21.305+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fde300608f0 con 0x7fde1806ae70 2024-06-01T00:48:21.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:21 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:21.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:21 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:22.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:22 smithi160 ceph-mon[36639]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:22.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:22 smithi160 ceph-mon[36639]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:48:22.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:22 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:48:22.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:22 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:48:22.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:22 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:22.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:22 smithi079 ceph-mon[29356]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:22.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:22 smithi079 ceph-mon[29356]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:48:22.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:22 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:48:22.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:22 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:48:22.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:22 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:24.466 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:24 smithi079 ceph-mon[29356]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:24.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:24 smithi160 ceph-mon[36639]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:26.113 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:25 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:26.114 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:25 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:26.114 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:25 smithi079 ceph-mon[29356]: Reconfiguring grafana.smithi079 (dependencies changed)... 2024-06-01T00:48:26.114 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:25 smithi079 ceph-mon[29356]: Reconfiguring daemon grafana.smithi079 on smithi079 2024-06-01T00:48:26.114 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:25 smithi079 ceph-mon[29356]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:26.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:25 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:26.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:25 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:26.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:25 smithi160 ceph-mon[36639]: Reconfiguring grafana.smithi079 (dependencies changed)... 2024-06-01T00:48:26.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:25 smithi160 ceph-mon[36639]: Reconfiguring daemon grafana.smithi079 on smithi079 2024-06-01T00:48:26.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:25 smithi160 ceph-mon[36639]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:27.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:26 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1248141751' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a46a931e-6c2b-457c-808f-90274056b4f3"}]: dispatch 2024-06-01T00:48:27.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:26 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1248141751' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a46a931e-6c2b-457c-808f-90274056b4f3"}]': finished 2024-06-01T00:48:27.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:26 smithi079 ceph-mon[29356]: osdmap e6: 1 total, 0 up, 1 in 2024-06-01T00:48:27.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:26 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:27.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:26 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1214361551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:48:27.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:26 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1248141751' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a46a931e-6c2b-457c-808f-90274056b4f3"}]: dispatch 2024-06-01T00:48:27.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:26 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1248141751' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a46a931e-6c2b-457c-808f-90274056b4f3"}]': finished 2024-06-01T00:48:27.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:26 smithi160 ceph-mon[36639]: osdmap e6: 1 total, 0 up, 1 in 2024-06-01T00:48:27.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:26 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:27.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:26 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1214361551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:48:28.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:27 smithi160 ceph-mon[36639]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:28.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:27 smithi079 ceph-mon[29356]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:30.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:30 smithi079 ceph-mon[29356]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:30.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:30 smithi160 ceph-mon[36639]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:32.524 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:32 smithi079 ceph-mon[29356]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:32.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:32 smithi160 ceph-mon[36639]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:34.431 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:34 smithi079 ceph-mon[29356]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:34.762 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:34 smithi160 ceph-mon[36639]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:36.045 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:36.045 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:36.046 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:35 smithi079 ceph-mon[29356]: Reconfiguring prometheus.smithi079 (dependencies changed)... 2024-06-01T00:48:36.046 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:35 smithi079 ceph-mon[29356]: Reconfiguring daemon prometheus.smithi079 on smithi079 2024-06-01T00:48:36.046 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:35 smithi079 ceph-mon[29356]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:36.046 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:36.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:36.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:36.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:35 smithi160 ceph-mon[36639]: Reconfiguring prometheus.smithi079 (dependencies changed)... 2024-06-01T00:48:36.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:35 smithi160 ceph-mon[36639]: Reconfiguring daemon prometheus.smithi079 on smithi079 2024-06-01T00:48:36.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:35 smithi160 ceph-mon[36639]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:36.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:37.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:37 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T00:48:37.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:37 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:37.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:37 smithi160 ceph-mon[36639]: Deploying daemon osd.0 on smithi079 2024-06-01T00:48:37.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:37 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T00:48:37.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:37 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:37.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:37 smithi079 ceph-mon[29356]: Deploying daemon osd.0 on smithi079 2024-06-01T00:48:38.100 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:38 smithi079 ceph-mon[29356]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:38.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:38 smithi160 ceph-mon[36639]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:40.472 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:40 smithi079 ceph-mon[29356]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:40.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:40 smithi160 ceph-mon[36639]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:41.976 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:41.977 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:41.977 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:41 smithi079 ceph-mon[29356]: Reconfiguring ceph-exporter.smithi160 (monmap changed)... 2024-06-01T00:48:41.977 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-01T00:48:41.977 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:41 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:41.977 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:41 smithi079 ceph-mon[29356]: Reconfiguring daemon ceph-exporter.smithi160 on smithi160 2024-06-01T00:48:41.977 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:41 smithi079 ceph-mon[29356]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:42.156 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:41 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:42.156 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:41 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:42.156 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:41 smithi160 ceph-mon[36639]: Reconfiguring ceph-exporter.smithi160 (monmap changed)... 2024-06-01T00:48:42.156 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:41 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi160", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-06-01T00:48:42.156 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:41 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:42.156 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:41 smithi160 ceph-mon[36639]: Reconfiguring daemon ceph-exporter.smithi160 on smithi160 2024-06-01T00:48:42.156 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:41 smithi160 ceph-mon[36639]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:43.291 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:43 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:43.291 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:43 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:43.291 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:43 smithi079 ceph-mon[29356]: Reconfiguring crash.smithi160 (monmap changed)... 2024-06-01T00:48:43.291 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:43 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-01T00:48:43.291 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:43 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:43.291 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:43 smithi079 ceph-mon[29356]: Reconfiguring daemon crash.smithi160 on smithi160 2024-06-01T00:48:43.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:43 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:43.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:43 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:43.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:43 smithi160 ceph-mon[36639]: Reconfiguring crash.smithi160 (monmap changed)... 2024-06-01T00:48:43.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:43 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi160", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-01T00:48:43.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:43 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:43.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:43 smithi160 ceph-mon[36639]: Reconfiguring daemon crash.smithi160 on smithi160 2024-06-01T00:48:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: Reconfiguring mgr.smithi160.dxcuke (monmap changed)... 2024-06-01T00:48:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.dxcuke", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T00:48:44.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T00:48:44.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:44.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:44 smithi079 ceph-mon[29356]: Reconfiguring daemon mgr.smithi160.dxcuke on smithi160 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: Reconfiguring mgr.smithi160.dxcuke (monmap changed)... 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi160.dxcuke", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:44.686 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:44 smithi160 ceph-mon[36639]: Reconfiguring daemon mgr.smithi160.dxcuke on smithi160 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: Reconfiguring mon.smithi160 (monmap changed)... 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: Reconfiguring daemon mon.smithi160 on smithi160 2024-06-01T00:48:45.829 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:45 smithi079 ceph-mon[29356]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:45.983 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:45.983 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:45.983 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: Reconfiguring mon.smithi160 (monmap changed)... 2024-06-01T00:48:45.983 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T00:48:45.983 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T00:48:45.983 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:45.983 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: Reconfiguring daemon mon.smithi160 on smithi160 2024-06-01T00:48:45.984 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:45 smithi160 ceph-mon[36639]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.247 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T00:48:47.288 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T00:48:47.289 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.289 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T00:48:47.289 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T00:48:47.289 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T00:48:47.289 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T00:48:47.289 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:47.289 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:48:48.142 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:47 smithi079 ceph-mon[29356]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:48.143 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:47 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:48.143 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:47 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:48.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:47 smithi160 ceph-mon[36639]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:48.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:47 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:48.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:47 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.942+0000 7fde2d7fa640 1 -- 172.21.15.79:0/553047844 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fde300608f0 con 0x7fde1806ae70 2024-06-01T00:48:48.944 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 0 on host 'smithi079' 2024-06-01T00:48:48.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde1806ae70 msgr2=0x7fde1806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:48.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde1806ae70 0x7fde1806d330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde20009230 tx=0x7fde20005b50 comp rx=0 tx=0).stop 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30101490 msgr2=0x7fde300728a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30101490 0x7fde300728a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fde2400b420 tx=0x7fde2400b8e0 comp rx=0 tx=0).stop 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 shutdown_connections 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde1806ae70 0x7fde1806d330 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fde30101490 0x7fde300728a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 --2- 172.21.15.79:0/553047844 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fde30100290 0x7fde30074200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 >> 172.21.15.79:0/553047844 conn(0x7fde300fb850 msgr2=0x7fde30104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 shutdown_connections 2024-06-01T00:48:48.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:48.944+0000 7fde36853640 1 -- 172.21.15.79:0/553047844 wait complete. 2024-06-01T00:48:49.933 DEBUG:teuthology.orchestra.run.smithi079:osd.0> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.0.service 2024-06-01T00:48:49.935 INFO:tasks.cephadm:Deploying osd.1 on smithi079 with /dev/vg_nvme/lv_3... 2024-06-01T00:48:49.936 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-01T00:48:50.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:49 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:50.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:49 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:50.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:49 smithi160 ceph-mon[36639]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:50.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:49 smithi160 ceph-mon[36639]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T00:48:50.241 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:49 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:50.241 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:49 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:50.242 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:49 smithi079 ceph-mon[29356]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:50.242 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:49 smithi079 ceph-mon[29356]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T00:48:51.039 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:48:51.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:50 smithi160 ceph-mon[36639]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T00:48:51.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:50 smithi160 ceph-mon[36639]: osdmap e7: 1 total, 0 up, 1 in 2024-06-01T00:48:51.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:51.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:50 smithi160 ceph-mon[36639]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-06-01T00:48:51.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:51.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:50 smithi079 ceph-mon[29356]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T00:48:51.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:50 smithi079 ceph-mon[29356]: osdmap e7: 1 total, 0 up, 1 in 2024-06-01T00:48:51.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:51.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:50 smithi079 ceph-mon[29356]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-06-01T00:48:51.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:48:52.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-mon[29356]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-06-01T00:48:52.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-mon[29356]: osdmap e8: 1 total, 0 up, 1 in 2024-06-01T00:48:52.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:52.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:52.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-mon[29356]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:52.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-mon[29356]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' 2024-06-01T00:48:52.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0[47958]: 2024-06-01T00:48:51.771+0000 7f409c4b8640 -1 osd.0 0 waiting for initial osdmap 2024-06-01T00:48:52.107 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:48:51 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0[47958]: 2024-06-01T00:48:51.779+0000 7f40932b0640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T00:48:52.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:51 smithi160 ceph-mon[36639]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-06-01T00:48:52.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:51 smithi160 ceph-mon[36639]: osdmap e8: 1 total, 0 up, 1 in 2024-06-01T00:48:52.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:51 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:52.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:51 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:52.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:51 smithi160 ceph-mon[36639]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T00:48:52.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:51 smithi160 ceph-mon[36639]: from='osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789]' entity='osd.0' 2024-06-01T00:48:53.075 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:52 smithi079 ceph-mon[29356]: purged_snaps scrub starts 2024-06-01T00:48:53.075 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:52 smithi079 ceph-mon[29356]: purged_snaps scrub ok 2024-06-01T00:48:53.075 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:52 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:53.075 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:52 smithi079 ceph-mon[29356]: osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] boot 2024-06-01T00:48:53.076 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:52 smithi079 ceph-mon[29356]: osdmap e9: 1 total, 1 up, 1 in 2024-06-01T00:48:53.076 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:52 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:53.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:52 smithi160 ceph-mon[36639]: purged_snaps scrub starts 2024-06-01T00:48:53.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:52 smithi160 ceph-mon[36639]: purged_snaps scrub ok 2024-06-01T00:48:53.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:52 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:53.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:52 smithi160 ceph-mon[36639]: osd.0 [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] boot 2024-06-01T00:48:53.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:52 smithi160 ceph-mon[36639]: osdmap e9: 1 total, 1 up, 1 in 2024-06-01T00:48:53.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:52 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T00:48:53.982 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:53 smithi079 ceph-mon[29356]: pgmap v36: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:48:53.982 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:53 smithi079 ceph-mon[29356]: osdmap e10: 1 total, 1 up, 1 in 2024-06-01T00:48:54.110 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-01T00:48:54.201 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-01T00:48:54.215 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-06-01T00:48:54.215 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-06-01T00:48:54.215 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114247 s, 918 MB/s 2024-06-01T00:48:54.215 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-06-01T00:48:54.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:53 smithi160 ceph-mon[36639]: pgmap v36: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:48:54.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:53 smithi160 ceph-mon[36639]: osdmap e10: 1 total, 1 up, 1 in 2024-06-01T00:48:54.895 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2024-06-01T00:48:55.115 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:48:56.119 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:56.119 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:56.120 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:55 smithi079 ceph-mon[29356]: pgmap v38: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:48:56.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:55 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:56.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:55 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:48:56.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:55 smithi160 ceph-mon[36639]: pgmap v38: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:48:56.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 -- 172.21.15.79:0/4267256908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 msgr2=0x7f2948100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:56.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 --2- 172.21.15.79:0/4267256908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948100800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2934009880 tx=0x7f293402a4a0 comp rx=0 tx=0).stop 2024-06-01T00:48:56.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 -- 172.21.15.79:0/4267256908 shutdown_connections 2024-06-01T00:48:56.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 --2- 172.21.15.79:0/4267256908 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948100800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:56.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 --2- 172.21.15.79:0/4267256908 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29480ff180 0x7f29480ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:56.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 -- 172.21.15.79:0/4267256908 >> 172.21.15.79:0/4267256908 conn(0x7f29480fa700 msgr2=0x7f29480fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 -- 172.21.15.79:0/4267256908 shutdown_connections 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.579+0000 7f294ee8e640 1 -- 172.21.15.79:0/4267256908 wait complete. 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f294ee8e640 1 Processor -- start 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f294ee8e640 1 -- start start 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f294ee8e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29480ff180 0x7f2948194c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f294ee8e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948195150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f294ee8e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948195720 con 0x7f2948100380 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f294ee8e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948195890 con 0x7f29480ff180 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f2947fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948195150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f2947fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948195150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50930/0 (socket says 172.21.15.79:50930) 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f2947fff640 1 -- 172.21.15.79:0/3417731612 learned_addr learned my addr 172.21.15.79:0/3417731612 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.580+0000 7f294cc2a640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29480ff180 0x7f2948194c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f2947fff640 1 -- 172.21.15.79:0/3417731612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29480ff180 msgr2=0x7f2948194c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f2947fff640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29480ff180 0x7f2948194c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:48:56.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f2947fff640 1 -- 172.21.15.79:0/3417731612 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2934009560 con 0x7f2948100380 2024-06-01T00:48:56.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f294cc2a640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29480ff180 0x7f2948194c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:48:56.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f2947fff640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948195150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2934005520 tx=0x7f2934005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:56.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f2945ffb640 1 -- 172.21.15.79:0/3417731612 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2934004aa0 con 0x7f2948100380 2024-06-01T00:48:56.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294819a280 con 0x7f2948100380 2024-06-01T00:48:56.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f2945ffb640 1 -- 172.21.15.79:0/3417731612 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2934004c00 con 0x7f2948100380 2024-06-01T00:48:56.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f2945ffb640 1 -- 172.21.15.79:0/3417731612 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2934003b10 con 0x7f2948100380 2024-06-01T00:48:56.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.581+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294819a7f0 con 0x7f2948100380 2024-06-01T00:48:56.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.582+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294818ecf0 con 0x7f2948100380 2024-06-01T00:48:56.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.583+0000 7f2945ffb640 1 -- 172.21.15.79:0/3417731612 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7f2934003d40 con 0x7f2948100380 2024-06-01T00:48:56.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.583+0000 7f2945ffb640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f293806ae70 0x7f293806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:48:56.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.583+0000 7f294cc2a640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f293806ae70 0x7f293806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:48:56.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.583+0000 7f2945ffb640 1 -- 172.21.15.79:0/3417731612 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f29340aa2d0 con 0x7f2948100380 2024-06-01T00:48:56.585 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.583+0000 7f294cc2a640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f293806ae70 0x7f293806d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f293c0058c0 tx=0x7f293c005850 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:48:56.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.587+0000 7f2945ffb640 1 -- 172.21.15.79:0/3417731612 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f29340763f0 con 0x7f2948100380 2024-06-01T00:48:56.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:48:56.786+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f29480608f0 con 0x7f293806ae70 2024-06-01T00:48:57.076 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:48:57.076 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:48:57.076 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:57.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:48:57.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:48:57.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:48:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:57 smithi079 ceph-mon[29356]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:48:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:48:57 smithi079 ceph-mon[29356]: pgmap v39: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:48:58.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:57 smithi160 ceph-mon[36639]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:48:58.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:48:57 smithi160 ceph-mon[36639]: pgmap v39: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:00.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:00 smithi079 ceph-mon[29356]: pgmap v40: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:00.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:00 smithi160 ceph-mon[36639]: pgmap v40: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:01.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: Detected new or changed devices on smithi079 2024-06-01T00:49:01.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:01.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: Adjusting osd_memory_target on smithi079 to 11840M 2024-06-01T00:49:01.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:01.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:01.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:01.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:01 smithi079 ceph-mon[29356]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: Detected new or changed devices on smithi079 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: Adjusting osd_memory_target on smithi079 to 11840M 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:01.968 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:01.969 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:01.969 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:01 smithi160 ceph-mon[36639]: pgmap v41: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:03.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1614176395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba57b192-9f40-405c-b51e-2ce8229d54d0"}]: dispatch 2024-06-01T00:49:03.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1614176395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba57b192-9f40-405c-b51e-2ce8229d54d0"}]': finished 2024-06-01T00:49:03.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:02 smithi160 ceph-mon[36639]: osdmap e11: 2 total, 1 up, 2 in 2024-06-01T00:49:03.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:03.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:03.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:03.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/749227260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:49:03.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:02 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1614176395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba57b192-9f40-405c-b51e-2ce8229d54d0"}]: dispatch 2024-06-01T00:49:03.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:02 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1614176395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba57b192-9f40-405c-b51e-2ce8229d54d0"}]': finished 2024-06-01T00:49:03.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:02 smithi079 ceph-mon[29356]: osdmap e11: 2 total, 1 up, 2 in 2024-06-01T00:49:03.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:02 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:03.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:02 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:03.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:02 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:03.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:02 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/749227260' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:49:03.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:03 smithi079 ceph-mon[29356]: pgmap v43: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:04.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:03 smithi160 ceph-mon[36639]: pgmap v43: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:06.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:06.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:06.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:06.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:06.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:06.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:06 smithi079 ceph-mon[29356]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:06.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:06.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:06.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:06.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:06.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:06.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:06.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:06 smithi160 ceph-mon[36639]: pgmap v44: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:06.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:08.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:08 smithi079 ceph-mon[29356]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:08.623 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:08 smithi160 ceph-mon[36639]: pgmap v45: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:09.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:09 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T00:49:09.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:09 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:09.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:09 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T00:49:09.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:09 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:10.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:10 smithi079 ceph-mon[29356]: Deploying daemon osd.1 on smithi079 2024-06-01T00:49:10.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:10 smithi079 ceph-mon[29356]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:10.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:10 smithi160 ceph-mon[36639]: Deploying daemon osd.1 on smithi079 2024-06-01T00:49:10.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:10 smithi160 ceph-mon[36639]: pgmap v46: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:12.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:12 smithi079 ceph-mon[29356]: pgmap v47: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:12 smithi160 ceph-mon[36639]: pgmap v47: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:14.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:14 smithi079 ceph-mon[29356]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:14.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:14 smithi160 ceph-mon[36639]: pgmap v48: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:15.466 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:15.466 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:15.466 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:15.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:15.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:15.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:16.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:16 smithi079 ceph-mon[29356]: pgmap v49: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:16.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:16 smithi160 ceph-mon[36639]: pgmap v49: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:17.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:17 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:17.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:17 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:17.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:17 smithi160 ceph-mon[36639]: pgmap v50: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:17.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:17 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:17.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:17 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:17.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:17 smithi079 ceph-mon[29356]: pgmap v50: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:18.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.660+0000 7f2945ffb640 1 -- 172.21.15.79:0/3417731612 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f29480608f0 con 0x7f293806ae70 2024-06-01T00:49:18.662 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 1 on host 'smithi079' 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f293806ae70 msgr2=0x7f293806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f293806ae70 0x7f293806d330 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f293c0058c0 tx=0x7f293c005850 comp rx=0 tx=0).stop 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 msgr2=0x7f2948195150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948195150 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2934005520 tx=0x7f2934005550 comp rx=0 tx=0).stop 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 shutdown_connections 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f293806ae70 0x7f293806d330 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2948100380 0x7f2948195150 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 --2- 172.21.15.79:0/3417731612 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f29480ff180 0x7f2948194c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.663+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 >> 172.21.15.79:0/3417731612 conn(0x7f29480fa700 msgr2=0x7f29481039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.664+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 shutdown_connections 2024-06-01T00:49:18.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:18.664+0000 7f294ee8e640 1 -- 172.21.15.79:0/3417731612 wait complete. 2024-06-01T00:49:19.927 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:19 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:19.927 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:19 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:19.927 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:19 smithi079 ceph-mon[29356]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T00:49:19.927 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:19 smithi079 ceph-mon[29356]: pgmap v51: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:19.950 DEBUG:teuthology.orchestra.run.smithi079:osd.1> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.1.service 2024-06-01T00:49:19.953 INFO:tasks.cephadm:Deploying osd.2 on smithi079 with /dev/vg_nvme/lv_2... 2024-06-01T00:49:19.953 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-01T00:49:20.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:19 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:20.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:19 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:20.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:19 smithi160 ceph-mon[36639]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T00:49:20.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:19 smithi160 ceph-mon[36639]: pgmap v51: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:20.966 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:20 smithi079 ceph-mon[29356]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T00:49:20.966 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:20 smithi079 ceph-mon[29356]: osdmap e12: 2 total, 1 up, 2 in 2024-06-01T00:49:20.966 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:20.966 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:20 smithi079 ceph-mon[29356]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-06-01T00:49:20.966 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:21.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:20 smithi160 ceph-mon[36639]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T00:49:21.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:20 smithi160 ceph-mon[36639]: osdmap e12: 2 total, 1 up, 2 in 2024-06-01T00:49:21.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:21.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:20 smithi160 ceph-mon[36639]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-06-01T00:49:21.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:21.311 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:49:21.762 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-06-01T00:49:21.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: osdmap e13: 2 total, 1 up, 2 in 2024-06-01T00:49:21.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:21.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:21.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: pgmap v54: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:21.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:21.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:21.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-mon[29356]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' 2024-06-01T00:49:21.763 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1[55785]: 2024-06-01T00:49:21.498+0000 7f714485f640 -1 osd.1 0 waiting for initial osdmap 2024-06-01T00:49:21.764 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:21 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1[55785]: 2024-06-01T00:49:21.503+0000 7f713ce92640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: osdmap e13: 2 total, 1 up, 2 in 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: pgmap v54: 0 pgs: ; 0 B data, 30 MiB used, 89 GiB / 89 GiB avail 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:22.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:21 smithi160 ceph-mon[36639]: from='osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362]' entity='osd.1' 2024-06-01T00:49:22.979 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:22 smithi079 ceph-mon[29356]: purged_snaps scrub starts 2024-06-01T00:49:22.980 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:22 smithi079 ceph-mon[29356]: purged_snaps scrub ok 2024-06-01T00:49:22.980 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:22 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:22.980 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:22 smithi079 ceph-mon[29356]: osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] boot 2024-06-01T00:49:22.980 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:22 smithi079 ceph-mon[29356]: osdmap e14: 2 total, 2 up, 2 in 2024-06-01T00:49:22.980 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:22 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:23.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:22 smithi160 ceph-mon[36639]: purged_snaps scrub starts 2024-06-01T00:49:23.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:22 smithi160 ceph-mon[36639]: purged_snaps scrub ok 2024-06-01T00:49:23.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:22 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:23.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:22 smithi160 ceph-mon[36639]: osd.1 [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] boot 2024-06-01T00:49:23.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:22 smithi160 ceph-mon[36639]: osdmap e14: 2 total, 2 up, 2 in 2024-06-01T00:49:23.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:22 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T00:49:23.676 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-01T00:49:23.768 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-01T00:49:23.782 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-06-01T00:49:23.782 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-06-01T00:49:23.783 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124512 s, 842 MB/s 2024-06-01T00:49:23.783 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-06-01T00:49:23.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:23 smithi079 ceph-mon[29356]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:24.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:23 smithi160 ceph-mon[36639]: pgmap v56: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:24.406 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2024-06-01T00:49:24.816 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:24 smithi079 ceph-mon[29356]: osdmap e15: 2 total, 2 up, 2 in 2024-06-01T00:49:24.981 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:49:25.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:24 smithi160 ceph-mon[36639]: osdmap e15: 2 total, 2 up, 2 in 2024-06-01T00:49:25.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:25 smithi079 ceph-mon[29356]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:26.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:25 smithi160 ceph-mon[36639]: pgmap v58: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:27.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2378529764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 msgr2=0x7f3dc4105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:27.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2378529764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc4105a40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009880 tx=0x7f3db802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:49:27.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2378529764 shutdown_connections 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2378529764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc4105a40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2378529764 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3dc4100d20 0x7f3dc4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2378529764 >> 172.21.15.79:0/2378529764 conn(0x7f3dc40fa750 msgr2=0x7f3dc40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2378529764 shutdown_connections 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2378529764 wait complete. 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 Processor -- start 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.081+0000 7f3dca5c6640 1 -- start start 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dca5c6640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3dc4100d20 0x7f3dc4195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dca5c6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc41963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dca5c6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc41969a0 con 0x7f3dc4103650 2024-06-01T00:49:27.100 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dca5c6640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc4196b10 con 0x7f3dc4100d20 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc41963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc37fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc41963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52222/0 (socket says 172.21.15.79:52222) 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc37fe640 1 -- 172.21.15.79:0/2981693198 learned_addr learned my addr 172.21.15.79:0/2981693198 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc3fff640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3dc4100d20 0x7f3dc4195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc37fe640 1 -- 172.21.15.79:0/2981693198 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3dc4100d20 msgr2=0x7f3dc4195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc37fe640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3dc4100d20 0x7f3dc4195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc37fe640 1 -- 172.21.15.79:0/2981693198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db8009560 con 0x7f3dc4103650 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc3fff640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3dc4100d20 0x7f3dc4195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc37fe640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc41963d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3db8005610 tx=0x7f3db8005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc17fa640 1 -- 172.21.15.79:0/2981693198 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3db8004ba0 con 0x7f3dc4103650 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dc17fa640 1 -- 172.21.15.79:0/2981693198 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3db8004d00 con 0x7f3dc4103650 2024-06-01T00:49:27.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.082+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc419b4b0 con 0x7f3dc4103650 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.083+0000 7f3dc17fa640 1 -- 172.21.15.79:0/2981693198 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3db8003bd0 con 0x7f3dc4103650 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.083+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc419ba20 con 0x7f3dc4103650 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.084+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc4108ee0 con 0x7f3dc4103650 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.084+0000 7f3dc17fa640 1 -- 172.21.15.79:0/2981693198 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7f3db8031640 con 0x7f3dc4103650 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.084+0000 7f3dc17fa640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3dac06ae70 0x7f3dac06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.085+0000 7f3dc17fa640 1 -- 172.21.15.79:0/2981693198 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f3db80aa5a0 con 0x7f3dc4103650 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.085+0000 7f3dc3fff640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3dac06ae70 0x7f3dac06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.085+0000 7f3dc3fff640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3dac06ae70 0x7f3dac06d330 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009930 tx=0x7f3db4008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:49:27.102 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.089+0000 7f3dc17fa640 1 -- 172.21.15.79:0/2981693198 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3db8076510 con 0x7f3dc4103650 2024-06-01T00:49:27.290 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:27.289+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dc4104110 con 0x7f3dac06ae70 2024-06-01T00:49:27.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: Detected new or changed devices on smithi079 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: Adjusting osd_memory_target on smithi079 to 5920M 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:27 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: Detected new or changed devices on smithi079 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: Adjusting osd_memory_target on smithi079 to 5920M 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:27.559 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:27.560 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:27.560 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:27 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:28.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:28 smithi160 ceph-mon[36639]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:49:28.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:28 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:49:28.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:28 smithi160 ceph-mon[36639]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:28.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:28 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:49:28.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:28 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:28.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:28 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:28.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:28 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:28 smithi079 ceph-mon[29356]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:49:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:28 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:49:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:28 smithi079 ceph-mon[29356]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:28 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:49:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:28 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:28 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:28 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:30.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:30 smithi079 ceph-mon[29356]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:30.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:30 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/976629715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d75526b-189e-4b38-a3d2-8c26d77c70a9"}]: dispatch 2024-06-01T00:49:30.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:30 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/976629715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d75526b-189e-4b38-a3d2-8c26d77c70a9"}]': finished 2024-06-01T00:49:30.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:30 smithi079 ceph-mon[29356]: osdmap e16: 3 total, 2 up, 3 in 2024-06-01T00:49:30.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:30 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:30.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:30 smithi160 ceph-mon[36639]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:30.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:30 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/976629715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d75526b-189e-4b38-a3d2-8c26d77c70a9"}]: dispatch 2024-06-01T00:49:30.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:30 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/976629715' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d75526b-189e-4b38-a3d2-8c26d77c70a9"}]': finished 2024-06-01T00:49:30.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:30 smithi160 ceph-mon[36639]: osdmap e16: 3 total, 2 up, 3 in 2024-06-01T00:49:30.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:30 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:31.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:31 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3131179971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:49:31.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:31 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3131179971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:49:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:32 smithi160 ceph-mon[36639]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:32.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:32 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:32.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:32 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:32.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:32 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:32.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:32 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:32.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:32 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:32.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:32 smithi079 ceph-mon[29356]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:32.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:32 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:32.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:32 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:32.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:32 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:32.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:32 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:32.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:32 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:34.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:34 smithi160 ceph-mon[36639]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:34.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:34 smithi079 ceph-mon[29356]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:35.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:36 smithi160 ceph-mon[36639]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:36.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:36 smithi079 ceph-mon[29356]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:38 smithi079 ceph-mon[29356]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:38 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T00:49:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:38 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:38 smithi160 ceph-mon[36639]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:38.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:38 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T00:49:38.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:38 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:39.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:39 smithi160 ceph-mon[36639]: Deploying daemon osd.2 on smithi079 2024-06-01T00:49:39.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:39 smithi079 ceph-mon[29356]: Deploying daemon osd.2 on smithi079 2024-06-01T00:49:40.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:40 smithi079 ceph-mon[29356]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:40.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:40 smithi160 ceph-mon[36639]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:42.511 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:42 smithi079 ceph-mon[29356]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:42.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:42 smithi160 ceph-mon[36639]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:43.508 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:43 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:43.508 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:43 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:43.508 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:43 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:43.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:43 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:43.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:43 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:43.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:43 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:44 smithi079 ceph-mon[29356]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:44 smithi160 ceph-mon[36639]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:46.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.212+0000 7f3dc17fa640 1 -- 172.21.15.79:0/2981693198 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3dc4104110 con 0x7f3dac06ae70 2024-06-01T00:49:46.214 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 2 on host 'smithi079' 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3dac06ae70 msgr2=0x7f3dac06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3dac06ae70 0x7f3dac06d330 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009930 tx=0x7f3db4008040 comp rx=0 tx=0).stop 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 msgr2=0x7f3dc41963d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc41963d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3db8005610 tx=0x7f3db8005640 comp rx=0 tx=0).stop 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 shutdown_connections 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3dac06ae70 0x7f3dac06d330 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3dc4103650 0x7f3dc41963d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:46.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 --2- 172.21.15.79:0/2981693198 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3dc4100d20 0x7f3dc4195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:46.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 >> 172.21.15.79:0/2981693198 conn(0x7f3dc40fa750 msgr2=0x7f3dc40fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:49:46.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 shutdown_connections 2024-06-01T00:49:46.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:49:46.215+0000 7f3dca5c6640 1 -- 172.21.15.79:0/2981693198 wait complete. 2024-06-01T00:49:46.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:46 smithi160 ceph-mon[36639]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:46.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:46.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:46 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:46.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:46 smithi160 ceph-mon[36639]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T00:49:46.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:46 smithi079 ceph-mon[29356]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:46.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:46.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:46 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:46.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:46 smithi079 ceph-mon[29356]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T00:49:47.164 DEBUG:teuthology.orchestra.run.smithi079:osd.2> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.2.service 2024-06-01T00:49:47.166 INFO:tasks.cephadm:Deploying osd.3 on smithi160 with /dev/vg_nvme/lv_4... 2024-06-01T00:49:47.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-01T00:49:47.383 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:49:48.320 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:48 smithi160 ceph-mon[36639]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T00:49:48.320 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:48 smithi160 ceph-mon[36639]: osdmap e17: 3 total, 2 up, 3 in 2024-06-01T00:49:48.320 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:48 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:48.320 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:48 smithi160 ceph-mon[36639]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-06-01T00:49:48.320 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:48 smithi160 ceph-mon[36639]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:48.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:48 smithi079 ceph-mon[29356]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T00:49:48.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:48 smithi079 ceph-mon[29356]: osdmap e17: 3 total, 2 up, 3 in 2024-06-01T00:49:48.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:48 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:48.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:48 smithi079 ceph-mon[29356]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-06-01T00:49:48.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:48 smithi079 ceph-mon[29356]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:48.992 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-01T00:49:49.083 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-01T00:49:49.097 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-06-01T00:49:49.097 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-06-01T00:49:49.097 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113443 s, 924 MB/s 2024-06-01T00:49:49.097 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-06-01T00:49:49.140 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:49 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2[63407]: 2024-06-01T00:49:49.062+0000 7f11bdb7e640 -1 osd.2 0 waiting for initial osdmap 2024-06-01T00:49:49.140 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:49 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2[63407]: 2024-06-01T00:49:49.069+0000 7f11b61b1640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T00:49:49.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:49 smithi160 ceph-mon[36639]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-06-01T00:49:49.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:49 smithi160 ceph-mon[36639]: osdmap e18: 3 total, 2 up, 3 in 2024-06-01T00:49:49.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:49 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:49.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:49 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:49.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:49 smithi160 ceph-mon[36639]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' 2024-06-01T00:49:49.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:49 smithi079 ceph-mon[29356]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-06-01T00:49:49.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:49 smithi079 ceph-mon[29356]: osdmap e18: 3 total, 2 up, 3 in 2024-06-01T00:49:49.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:49 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:49.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:49 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:49.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:49 smithi079 ceph-mon[29356]: from='osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964]' entity='osd.2' 2024-06-01T00:49:49.684 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_4 2024-06-01T00:49:49.892 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:49:50.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: purged_snaps scrub starts 2024-06-01T00:49:50.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: purged_snaps scrub ok 2024-06-01T00:49:50.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:50.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:50.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:50.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:50.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] boot 2024-06-01T00:49:50.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: osdmap e19: 3 total, 3 up, 3 in 2024-06-01T00:49:50.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:50.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: purged_snaps scrub starts 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: purged_snaps scrub ok 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: osd.2 [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] boot 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: osdmap e19: 3 total, 3 up, 3 in 2024-06-01T00:49:50.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T00:49:51.386 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 -- 172.21.15.160:0/3568099649 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 msgr2=0x7fd860103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:51.386 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 --2- 172.21.15.160:0/3568099649 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860103190 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd850009880 tx=0x7fd85002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:49:51.386 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 -- 172.21.15.160:0/3568099649 shutdown_connections 2024-06-01T00:49:51.386 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 --2- 172.21.15.160:0/3568099649 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8601036d0 0x7fd860105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:51.387 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 --2- 172.21.15.160:0/3568099649 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860103190 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:51.387 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 -- 172.21.15.160:0/3568099649 >> 172.21.15.160:0/3568099649 conn(0x7fd8600fa750 msgr2=0x7fd8600fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:49:51.387 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 -- 172.21.15.160:0/3568099649 shutdown_connections 2024-06-01T00:49:51.387 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 -- 172.21.15.160:0/3568099649 wait complete. 2024-06-01T00:49:51.387 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 Processor -- start 2024-06-01T00:49:51.387 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.384+0000 7fd865e85640 1 -- start start 2024-06-01T00:49:51.389 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd865e85640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:49:51.389 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd865e85640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8601036d0 0x7fd8601962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:49:51.389 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd865e85640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601968a0 con 0x7fd8601036d0 2024-06-01T00:49:51.389 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd865e85640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860196a10 con 0x7fd860100da0 2024-06-01T00:49:51.389 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85f7fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85f7fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:47990/0 (socket says 172.21.15.160:47990) 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85f7fe640 1 -- 172.21.15.160:0/452601417 learned_addr learned my addr 172.21.15.160:0/452601417 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd856ffd640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8601036d0 0x7fd8601962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85f7fe640 1 -- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8601036d0 msgr2=0x7fd8601962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85f7fe640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8601036d0 0x7fd8601962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85f7fe640 1 -- 172.21.15.160:0/452601417 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd850009560 con 0x7fd860100da0 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85f7fe640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860195d90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd850005610 tx=0x7fd850005640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.385+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd850004ba0 con 0x7fd860100da0 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.386+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86019b3f0 con 0x7fd860100da0 2024-06-01T00:49:51.390 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.386+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd850004d00 con 0x7fd860100da0 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.386+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd850003bd0 con 0x7fd860100da0 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.386+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86019b960 con 0x7fd860100da0 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.387+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860108eb0 con 0x7fd860100da0 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.387+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 18) v1 ==== 89414+0+0 (secure 0 0 0) 0x7fd85003c470 con 0x7fd860100da0 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.387+0000 7fd85d7fa640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd84c06ae70 0x7fd84c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.387+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7fd8500aa5e0 con 0x7fd860100da0 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.388+0000 7fd856ffd640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd84c06ae70 0x7fd84c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:49:51.391 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.388+0000 7fd856ffd640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd84c06ae70 0x7fd84c06d330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd8480058c0 tx=0x7fd84800a250 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:49:51.393 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.392+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd850076540 con 0x7fd860100da0 2024-06-01T00:49:51.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:51 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:51.572 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:51.570+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8601040f0 con 0x7fd84c06ae70 2024-06-01T00:49:51.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:51 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:49:52.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:52 smithi160 ceph-mon[36639]: osdmap e20: 3 total, 3 up, 3 in 2024-06-01T00:49:52.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:52 smithi160 ceph-mon[36639]: pgmap v76: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:49:52.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:52 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-01T00:49:52.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:52 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:49:52.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:52 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:49:52.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:52 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:52.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:52 smithi079 ceph-mon[29356]: osdmap e20: 3 total, 3 up, 3 in 2024-06-01T00:49:52.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:52 smithi079 ceph-mon[29356]: pgmap v76: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:49:52.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:52 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-01T00:49:52.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:52 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:49:52.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:52 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:49:52.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:52 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:53.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:49:53.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-01T00:49:53.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: osdmap e21: 3 total, 3 up, 3 in 2024-06-01T00:49:53.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-01T00:49:53.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.556 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:53 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:53.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:49:53.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-01T00:49:53.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: osdmap e21: 3 total, 3 up, 3 in 2024-06-01T00:49:53.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-01T00:49:53.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:53.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:53 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: Detected new or changed devices on smithi079 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: Adjusting osd_memory_target on smithi079 to 3946M 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: osdmap e22: 3 total, 3 up, 3 in 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/473330660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b28a6b8-616c-4949-9ef8-8197138f624f"}]: dispatch 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b28a6b8-616c-4949-9ef8-8197138f624f"}]: dispatch 2024-06-01T00:49:54.337 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b28a6b8-616c-4949-9ef8-8197138f624f"}]': finished 2024-06-01T00:49:54.338 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: osdmap e23: 4 total, 3 up, 4 in 2024-06-01T00:49:54.338 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:54 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:49:54.449 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: Detected new or changed devices on smithi079 2024-06-01T00:49:54.449 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: Adjusting osd_memory_target on smithi079 to 3946M 2024-06-01T00:49:54.449 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-01T00:49:54.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: osdmap e22: 3 total, 3 up, 3 in 2024-06-01T00:49:54.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:49:54.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/473330660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b28a6b8-616c-4949-9ef8-8197138f624f"}]: dispatch 2024-06-01T00:49:54.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b28a6b8-616c-4949-9ef8-8197138f624f"}]: dispatch 2024-06-01T00:49:54.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b28a6b8-616c-4949-9ef8-8197138f624f"}]': finished 2024-06-01T00:49:54.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: osdmap e23: 4 total, 3 up, 4 in 2024-06-01T00:49:54.450 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:54 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:49:54.597 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68143]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T00:49:54.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68143]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:54.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68143]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:54.598 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68143]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:54.855 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68188]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T00:49:54.856 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68188]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:54.856 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68188]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:54.856 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68188]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:55.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68217]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T00:49:55.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68217]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:55.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68217]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:55.164 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:54 smithi079 sudo[68217]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:55.424 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/2511456362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:49:55.424 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:55.424 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 ceph-mon[29356]: osdmap e24: 4 total, 3 up, 4 in 2024-06-01T00:49:55.424 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:49:55.424 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T00:49:55.424 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:55.424 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68221]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:55.424 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68221]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:55.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:55 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/2511456362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:49:55.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:55 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:55.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:55 smithi160 ceph-mon[36639]: osdmap e24: 4 total, 3 up, 4 in 2024-06-01T00:49:55.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:55 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:49:55.693 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68269]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T00:49:55.694 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68269]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:55.694 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68269]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:55.694 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68269]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:55.954 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68304]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T00:49:55.955 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:55.955 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68304]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:55.955 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68304]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:55.955 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68301]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T00:49:55.955 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68301]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:55.955 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68301]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:55.955 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:49:55 smithi079 sudo[68301]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:56.542 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 sudo[68351]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T00:49:56.542 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 sudo[68351]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:56.542 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 sudo[68351]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:56.542 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 sudo[68351]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:56.715 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41249]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T00:49:56.715 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41249]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:56.715 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41249]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:56.715 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41249]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:56.729 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:49:56.727+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd850003d30 con 0x7fd860100da0 2024-06-01T00:49:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 ceph-mon[29356]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:49:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 ceph-mon[29356]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T00:49:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 ceph-mon[29356]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T00:49:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:49:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:49:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:49:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:49:57.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41255]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41255]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41255]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 sudo[41255]: pam_unix(sudo:session): session closed for user root 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 ceph-mon[36639]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 ceph-mon[36639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 ceph-mon[36639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-06-01T00:49:57.055 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "mon metadata", "id": "smithi160"}]: dispatch 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: mgrmap e19: smithi079.emylav(active, since 2m), standbys: smithi160.dxcuke 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:58.004 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.005 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:49:57 smithi160 ceph-mon[36639]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: mgrmap e19: smithi079.emylav(active, since 2m), standbys: smithi160.dxcuke 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:49:58.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:49:58.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:49:58.107 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:49:57 smithi079 ceph-mon[29356]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:00.516 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:00 smithi160 ceph-mon[36639]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:00.517 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:00 smithi160 ceph-mon[36639]: overall HEALTH_WARN 1 mgr modules have recently crashed 2024-06-01T00:50:00.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:00 smithi079 ceph-mon[29356]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:00.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:00 smithi079 ceph-mon[29356]: overall HEALTH_WARN 1 mgr modules have recently crashed 2024-06-01T00:50:01.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T00:50:01.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:01.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T00:50:01.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:02.420 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:02 smithi160 ceph-mon[36639]: Deploying daemon osd.3 on smithi160 2024-06-01T00:50:02.420 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:02 smithi160 ceph-mon[36639]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:02 smithi079 ceph-mon[29356]: Deploying daemon osd.3 on smithi160 2024-06-01T00:50:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:02 smithi079 ceph-mon[29356]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:04.669 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:04 smithi160 ceph-mon[36639]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:04.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:04 smithi079 ceph-mon[29356]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:05.662 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:05.778 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:06.635 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:06 smithi160 ceph-mon[36639]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:06.635 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:06.635 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:06.635 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:06 smithi079 ceph-mon[29356]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:08.413 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:08 smithi160 ceph-mon[36639]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:08.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:08 smithi079 ceph-mon[29356]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:09.585 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 3 on host 'smithi160' 2024-06-01T00:50:09.585 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.583+0000 7fd85d7fa640 1 -- 172.21.15.160:0/452601417 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd8601040f0 con 0x7fd84c06ae70 2024-06-01T00:50:09.587 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd84c06ae70 msgr2=0x7fd84c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:09.587 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd84c06ae70 0x7fd84c06d330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd8480058c0 tx=0x7fd84800a250 comp rx=0 tx=0).stop 2024-06-01T00:50:09.587 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 msgr2=0x7fd860195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:09.587 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860195d90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd850005610 tx=0x7fd850005640 comp rx=0 tx=0).stop 2024-06-01T00:50:09.587 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 shutdown_connections 2024-06-01T00:50:09.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd84c06ae70 0x7fd84c06d330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:09.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd8601036d0 0x7fd8601962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:09.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 --2- 172.21.15.160:0/452601417 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd860100da0 0x7fd860195d90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:09.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 >> 172.21.15.160:0/452601417 conn(0x7fd8600fa750 msgr2=0x7fd8600fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:50:09.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 shutdown_connections 2024-06-01T00:50:09.588 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:09.586+0000 7fd865e85640 1 -- 172.21.15.160:0/452601417 wait complete. 2024-06-01T00:50:10.176 DEBUG:teuthology.orchestra.run.smithi160:osd.3> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.3.service 2024-06-01T00:50:10.178 INFO:tasks.cephadm:Deploying osd.4 on smithi160 with /dev/vg_nvme/lv_3... 2024-06-01T00:50:10.179 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-01T00:50:10.433 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:50:10.549 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:10 smithi160 ceph-mon[36639]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:10.550 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:10.550 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:10.550 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:10 smithi160 ceph-mon[36639]: from='osd.3 [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T00:50:10.550 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:10 smithi160 ceph-mon[36639]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T00:50:10.550 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:10.550 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:10 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:10.840 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:10 smithi079 ceph-mon[29356]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:10.840 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:10.840 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:10.840 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:10 smithi079 ceph-mon[29356]: from='osd.3 [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T00:50:10.840 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:10 smithi079 ceph-mon[29356]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T00:50:10.840 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:10.840 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:10 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:11.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:11 smithi079 ceph-mon[29356]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T00:50:11.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:11 smithi079 ceph-mon[29356]: osdmap e25: 4 total, 3 up, 4 in 2024-06-01T00:50:11.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:11 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:11.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:11 smithi079 ceph-mon[29356]: from='osd.3 [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:11.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:11 smithi079 ceph-mon[29356]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:11.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:11 smithi079 ceph-mon[29356]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:11.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-mon[36639]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T00:50:11.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-mon[36639]: osdmap e25: 4 total, 3 up, 4 in 2024-06-01T00:50:11.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:11.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-mon[36639]: from='osd.3 [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:11.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-mon[36639]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:11.861 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-mon[36639]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-06-01T00:50:11.862 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3[42480]: 2024-06-01T00:50:11.600+0000 7f21b182c640 -1 osd.3 0 waiting for initial osdmap 2024-06-01T00:50:11.862 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:50:11 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3[42480]: 2024-06-01T00:50:11.606+0000 7f21ade67640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T00:50:12.686 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-01T00:50:12.686 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-01T00:50:12.686 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-06-01T00:50:12.686 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-06-01T00:50:12.686 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124512 s, 842 MB/s 2024-06-01T00:50:12.686 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-06-01T00:50:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:12 smithi160 ceph-mon[36639]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-06-01T00:50:12.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:12 smithi160 ceph-mon[36639]: osdmap e26: 4 total, 3 up, 4 in 2024-06-01T00:50:12.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:12 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:12.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:12 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:12.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:12 smithi079 ceph-mon[29356]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-06-01T00:50:12.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:12 smithi079 ceph-mon[29356]: osdmap e26: 4 total, 3 up, 4 in 2024-06-01T00:50:12.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:12 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:12.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:12 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:13.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:13 smithi160 ceph-mon[36639]: purged_snaps scrub starts 2024-06-01T00:50:13.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:13 smithi160 ceph-mon[36639]: purged_snaps scrub ok 2024-06-01T00:50:13.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:13 smithi160 ceph-mon[36639]: osd.3 [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] boot 2024-06-01T00:50:13.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:13 smithi160 ceph-mon[36639]: osdmap e27: 4 total, 4 up, 4 in 2024-06-01T00:50:13.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:13 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:13.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:13 smithi160 ceph-mon[36639]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:13.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:13 smithi079 ceph-mon[29356]: purged_snaps scrub starts 2024-06-01T00:50:13.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:13 smithi079 ceph-mon[29356]: purged_snaps scrub ok 2024-06-01T00:50:13.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:13 smithi079 ceph-mon[29356]: osd.3 [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] boot 2024-06-01T00:50:13.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:13 smithi079 ceph-mon[29356]: osdmap e27: 4 total, 4 up, 4 in 2024-06-01T00:50:13.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:13 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T00:50:13.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:13 smithi079 ceph-mon[29356]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:13.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_3 2024-06-01T00:50:14.222 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:50:14.743 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:14 smithi160 ceph-mon[36639]: osdmap e28: 4 total, 4 up, 4 in 2024-06-01T00:50:14.951 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:14 smithi079 ceph-mon[29356]: osdmap e28: 4 total, 4 up, 4 in 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: osdmap e29: 4 total, 4 up, 4 in 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: Detected new or changed devices on smithi160 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: Adjusting osd_memory_target on smithi160 to 14912M 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:15.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:15.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:15 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: osdmap e29: 4 total, 4 up, 4 in 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: Detected new or changed devices on smithi160 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: Adjusting osd_memory_target on smithi160 to 14912M 2024-06-01T00:50:15.900 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:15.901 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:15.901 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:15.901 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:15 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:16.441 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 -- 172.21.15.160:0/322830528 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1002c0 msgr2=0x7ffb5c1006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:16.441 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/322830528 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1002c0 0x7ffb5c1006e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009880 tx=0x7ffb4c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:50:16.441 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 -- 172.21.15.160:0/322830528 shutdown_connections 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/322830528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb5c1014c0 0x7ffb5c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/322830528 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1002c0 0x7ffb5c1006e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 -- 172.21.15.160:0/322830528 >> 172.21.15.160:0/322830528 conn(0x7ffb5c0fb860 msgr2=0x7ffb5c0fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 -- 172.21.15.160:0/322830528 shutdown_connections 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.434+0000 7ffb62a0f640 1 -- 172.21.15.160:0/322830528 wait complete. 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb62a0f640 1 Processor -- start 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb62a0f640 1 -- start start 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb62a0f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb5c1002c0 0x7ffb5c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb62a0f640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1014c0 0x7ffb5c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:50:16.442 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb62a0f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c1968f0 con 0x7ffb5c1002c0 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb62a0f640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c196a60 con 0x7ffb5c1014c0 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb5b7fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1014c0 0x7ffb5c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb5b7fe640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1014c0 0x7ffb5c196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:52266/0 (socket says 172.21.15.160:52266) 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.435+0000 7ffb5b7fe640 1 -- 172.21.15.160:0/1267446487 learned_addr learned my addr 172.21.15.160:0/1267446487 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb5bfff640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb5c1002c0 0x7ffb5c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb5b7fe640 1 -- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb5c1002c0 msgr2=0x7ffb5c195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb5b7fe640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb5c1002c0 0x7ffb5c195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb5b7fe640 1 -- 172.21.15.160:0/1267446487 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c009560 con 0x7ffb5c1014c0 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb5b7fe640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1014c0 0x7ffb5c196320 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000d6f0 tx=0x7ffb5000dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb597fa640 1 -- 172.21.15.160:0/1267446487 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb50009a30 con 0x7ffb5c1014c0 2024-06-01T00:50:16.443 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb597fa640 1 -- 172.21.15.160:0/1267446487 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb500054e0 con 0x7ffb5c1014c0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c19b450 con 0x7ffb5c1014c0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb597fa640 1 -- 172.21.15.160:0/1267446487 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb50010460 con 0x7ffb5c1014c0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.436+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c19ba20 con 0x7ffb5c1014c0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.437+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c18fe50 con 0x7ffb5c1014c0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.437+0000 7ffb597fa640 1 -- 172.21.15.160:0/1267446487 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ffb50009b90 con 0x7ffb5c1014c0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.438+0000 7ffb597fa640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ffb4406aef0 0x7ffb4406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.438+0000 7ffb597fa640 1 -- 172.21.15.160:0/1267446487 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7ffb50089440 con 0x7ffb5c1014c0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.438+0000 7ffb5bfff640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ffb4406aef0 0x7ffb4406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:50:16.444 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.438+0000 7ffb5bfff640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ffb4406aef0 0x7ffb4406d3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c197220 tx=0x7ffb4c005d50 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:50:16.445 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.442+0000 7ffb597fa640 1 -- 172.21.15.160:0/1267446487 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ffb50054d80 con 0x7ffb5c1014c0 2024-06-01T00:50:16.624 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:16.624+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb5c02c4f0 con 0x7ffb4406aef0 2024-06-01T00:50:16.987 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:16.987 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:16 smithi160 ceph-mon[36639]: osdmap e30: 4 total, 4 up, 4 in 2024-06-01T00:50:16.987 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:16.987 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:16.987 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:16 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:17.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:17.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:16 smithi079 ceph-mon[29356]: osdmap e30: 4 total, 4 up, 4 in 2024-06-01T00:50:17.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:17.017 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:17.018 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:16 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:17.881 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:17 smithi160 ceph-mon[36639]: osdmap e31: 4 total, 4 up, 4 in 2024-06-01T00:50:17.881 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:17 smithi160 ceph-mon[36639]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:50:17.881 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:17 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:50:17.881 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:17 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:50:17.881 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:17 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:17.881 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:17 smithi160 ceph-mon[36639]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:18.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:17 smithi079 ceph-mon[29356]: osdmap e31: 4 total, 4 up, 4 in 2024-06-01T00:50:18.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:17 smithi079 ceph-mon[29356]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:50:18.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:17 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:50:18.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:17 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:50:18.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:17 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:18.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:17 smithi079 ceph-mon[29356]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:19.256 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:18 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/3077852824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb728d-8dc0-4629-a8ef-03def9adf728"}]: dispatch 2024-06-01T00:50:19.256 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:18 smithi160 ceph-mon[36639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb728d-8dc0-4629-a8ef-03def9adf728"}]: dispatch 2024-06-01T00:50:19.256 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:18 smithi160 ceph-mon[36639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adb728d-8dc0-4629-a8ef-03def9adf728"}]': finished 2024-06-01T00:50:19.256 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:18 smithi160 ceph-mon[36639]: osdmap e32: 5 total, 4 up, 5 in 2024-06-01T00:50:19.256 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:18 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:19.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:18 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/3077852824' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb728d-8dc0-4629-a8ef-03def9adf728"}]: dispatch 2024-06-01T00:50:19.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:18 smithi079 ceph-mon[29356]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb728d-8dc0-4629-a8ef-03def9adf728"}]: dispatch 2024-06-01T00:50:19.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:18 smithi079 ceph-mon[29356]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adb728d-8dc0-4629-a8ef-03def9adf728"}]': finished 2024-06-01T00:50:19.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:18 smithi079 ceph-mon[29356]: osdmap e32: 5 total, 4 up, 5 in 2024-06-01T00:50:19.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:18 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:20.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:19 smithi160 ceph-mon[36639]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-06-01T00:50:20.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:19 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1748701503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:50:20.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:19 smithi079 ceph-mon[29356]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-06-01T00:50:20.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:19 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1748701503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:50:21.304 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:21.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:22.305 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:21 smithi160 ceph-mon[36639]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-01T00:50:22.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:21 smithi079 ceph-mon[29356]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-01T00:50:24.802 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:24 smithi160 ceph-mon[36639]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-01T00:50:24.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:24 smithi079 ceph-mon[29356]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-01T00:50:26.638 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:26 smithi160 ceph-mon[36639]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-06-01T00:50:26.638 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:26 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T00:50:26.638 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:26 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:26.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:26 smithi079 ceph-mon[29356]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-06-01T00:50:26.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:26 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T00:50:26.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:26 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:27.664 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:27 smithi160 ceph-mon[36639]: Deploying daemon osd.4 on smithi160 2024-06-01T00:50:27.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:27 smithi079 ceph-mon[29356]: Deploying daemon osd.4 on smithi160 2024-06-01T00:50:28.683 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:28 smithi160 ceph-mon[36639]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-01T00:50:28.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:28 smithi079 ceph-mon[29356]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-01T00:50:30.483 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:30 smithi160 ceph-mon[36639]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:30.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:30 smithi079 ceph-mon[29356]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:31.631 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:31 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:31.631 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:31 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:31.631 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:31 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:31.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:31 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:31.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:31 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:31.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:31 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:32.653 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:32 smithi160 ceph-mon[36639]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:32.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:32 smithi079 ceph-mon[29356]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:34.363 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.362+0000 7ffb597fa640 1 -- 172.21.15.160:0/1267446487 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffb5c02c4f0 con 0x7ffb4406aef0 2024-06-01T00:50:34.363 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 4 on host 'smithi160' 2024-06-01T00:50:34.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ffb4406aef0 msgr2=0x7ffb4406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:34.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ffb4406aef0 0x7ffb4406d3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c197220 tx=0x7ffb4c005d50 comp rx=0 tx=0).stop 2024-06-01T00:50:34.365 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1014c0 msgr2=0x7ffb5c196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1014c0 0x7ffb5c196320 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000d6f0 tx=0x7ffb5000dbb0 comp rx=0 tx=0).stop 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 shutdown_connections 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ffb4406aef0 0x7ffb4406d3b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ffb5c1014c0 0x7ffb5c196320 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 --2- 172.21.15.160:0/1267446487 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb5c1002c0 0x7ffb5c195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 >> 172.21.15.160:0/1267446487 conn(0x7ffb5c0fb860 msgr2=0x7ffb5c104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 shutdown_connections 2024-06-01T00:50:34.366 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:34.364+0000 7ffb62a0f640 1 -- 172.21.15.160:0/1267446487 wait complete. 2024-06-01T00:50:34.704 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:34 smithi160 ceph-mon[36639]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:34.704 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:34 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:34.704 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:34 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:34.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:34 smithi079 ceph-mon[29356]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:34.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:34 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:34.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:34 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:35.318 DEBUG:teuthology.orchestra.run.smithi160:osd.4> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.4.service 2024-06-01T00:50:35.321 INFO:tasks.cephadm:Deploying osd.5 on smithi160 with /dev/vg_nvme/lv_2... 2024-06-01T00:50:35.321 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-01T00:50:35.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:35 smithi160 ceph-mon[36639]: from='osd.4 [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T00:50:35.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:35 smithi160 ceph-mon[36639]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T00:50:35.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:35.583 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:50:35.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:35 smithi079 ceph-mon[29356]: from='osd.4 [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T00:50:35.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:35 smithi079 ceph-mon[29356]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T00:50:35.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:36.390 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:36.391 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: osdmap e33: 5 total, 4 up, 5 in 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: from='osd.4 [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:36.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:36.556 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:50:36.393+0000 7feeaed63640 -1 osd.4 0 waiting for initial osdmap 2024-06-01T00:50:36.556 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:50:36 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:50:36.400+0000 7feea9b63640 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T00:50:36.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: osdmap e33: 5 total, 4 up, 5 in 2024-06-01T00:50:36.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:36.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: from='osd.4 [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:36.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:50:36.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:36.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:36 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:37.227 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-01T00:50:37.307 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-01T00:50:37.323 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-06-01T00:50:37.323 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-06-01T00:50:37.323 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0143725 s, 730 MB/s 2024-06-01T00:50:37.323 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-06-01T00:50:37.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:37 smithi160 ceph-mon[36639]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-06-01T00:50:37.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:37 smithi160 ceph-mon[36639]: osdmap e34: 5 total, 4 up, 5 in 2024-06-01T00:50:37.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:37 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:37.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:37 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:37.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:37 smithi079 ceph-mon[29356]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-06-01T00:50:37.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:37 smithi079 ceph-mon[29356]: osdmap e34: 5 total, 4 up, 5 in 2024-06-01T00:50:37.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:37 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:37.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:37 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:38.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi160:vg_nvme/lv_2 2024-06-01T00:50:38.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:38 smithi160 ceph-mon[36639]: purged_snaps scrub starts 2024-06-01T00:50:38.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:38 smithi160 ceph-mon[36639]: purged_snaps scrub ok 2024-06-01T00:50:38.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:38 smithi160 ceph-mon[36639]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:38.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:38 smithi160 ceph-mon[36639]: osd.4 [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] boot 2024-06-01T00:50:38.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:38 smithi160 ceph-mon[36639]: osdmap e35: 5 total, 5 up, 5 in 2024-06-01T00:50:38.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:38 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:38.809 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:50:38.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:38 smithi079 ceph-mon[29356]: purged_snaps scrub starts 2024-06-01T00:50:38.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:38 smithi079 ceph-mon[29356]: purged_snaps scrub ok 2024-06-01T00:50:38.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:38 smithi079 ceph-mon[29356]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-06-01T00:50:38.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:38 smithi079 ceph-mon[29356]: osd.4 [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] boot 2024-06-01T00:50:38.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:38 smithi079 ceph-mon[29356]: osdmap e35: 5 total, 5 up, 5 in 2024-06-01T00:50:38.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:38 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T00:50:39.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:39 smithi160 ceph-mon[36639]: osdmap e36: 5 total, 5 up, 5 in 2024-06-01T00:50:39.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:39 smithi079 ceph-mon[29356]: osdmap e36: 5 total, 5 up, 5 in 2024-06-01T00:50:40.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:40 smithi160 ceph-mon[36639]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:40.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:40 smithi079 ceph-mon[29356]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:41.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 -- 172.21.15.160:0/1158418611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 msgr2=0x7fdea0100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:41.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/1158418611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea0100800 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde94009c20 tx=0x7fde9402a970 comp rx=0 tx=0).stop 2024-06-01T00:50:41.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 -- 172.21.15.160:0/1158418611 shutdown_connections 2024-06-01T00:50:41.549 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/1158418611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea0100800 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/1158418611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea00ff180 0x7fdea00ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 -- 172.21.15.160:0/1158418611 >> 172.21.15.160:0/1158418611 conn(0x7fdea00fa720 msgr2=0x7fdea00fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 -- 172.21.15.160:0/1158418611 shutdown_connections 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.540+0000 7fdea5b0e640 1 -- 172.21.15.160:0/1158418611 wait complete. 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.541+0000 7fdea5b0e640 1 Processor -- start 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.541+0000 7fdea5b0e640 1 -- start start 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.541+0000 7fdea5b0e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea00ff180 0x7fdea0069210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fdea5b0e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fdea5b0e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea019b190 con 0x7fdea00ff180 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fdea5b0e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea019b2d0 con 0x7fdea0100380 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9effd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:50:41.550 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9effd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea019ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:57106/0 (socket says 172.21.15.160:57106) 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9effd640 1 -- 172.21.15.160:0/4248815034 learned_addr learned my addr 172.21.15.160:0/4248815034 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9f7fe640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea00ff180 0x7fdea0069210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9effd640 1 -- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea00ff180 msgr2=0x7fdea0069210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9effd640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea00ff180 0x7fdea0069210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9effd640 1 -- 172.21.15.160:0/4248815034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde94009900 con 0x7fdea0100380 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9effd640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea019ac50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde9402fd20 tx=0x7fde9402f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9cff9640 1 -- 172.21.15.160:0/4248815034 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde94030840 con 0x7fdea0100380 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9cff9640 1 -- 172.21.15.160:0/4248815034 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde94004040 con 0x7fdea0100380 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea019b4d0 con 0x7fdea0100380 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fde9cff9640 1 -- 172.21.15.160:0/4248815034 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde94039680 con 0x7fdea0100380 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.542+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea019ba40 con 0x7fdea0100380 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.543+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea018ed90 con 0x7fdea0100380 2024-06-01T00:50:41.551 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.544+0000 7fde9cff9640 1 -- 172.21.15.160:0/4248815034 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fde9400a040 con 0x7fdea0100380 2024-06-01T00:50:41.552 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.544+0000 7fde9cff9640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde8806af00 0x7fde8806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:50:41.552 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.545+0000 7fde9f7fe640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde8806af00 0x7fde8806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:50:41.552 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.545+0000 7fde9cff9640 1 -- 172.21.15.160:0/4248815034 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7fde9402c040 con 0x7fdea0100380 2024-06-01T00:50:41.552 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.545+0000 7fde9f7fe640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde8806af00 0x7fde8806d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdea01001e0 tx=0x7fde90008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:50:41.552 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.549+0000 7fde9cff9640 1 -- 172.21.15.160:0/4248815034 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fde94076800 con 0x7fdea0100380 2024-06-01T00:50:41.730 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:41.729+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdea002c450 con 0x7fde8806af00 2024-06-01T00:50:42.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: Detected new or changed devices on smithi160 2024-06-01T00:50:42.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:50:42.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:50:42.356 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: Adjusting osd_memory_target on smithi160 to 7456M 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:50:42.357 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:42 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:42.377 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: Detected new or changed devices on smithi160 2024-06-01T00:50:42.377 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.377 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.377 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:50:42.377 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:50:42.377 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: Adjusting osd_memory_target on smithi160 to 7456M 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T00:50:42.378 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:42 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:43.441 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:43 smithi160 ceph-mon[36639]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:50:43.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:43 smithi079 ceph-mon[29356]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:50:44.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:44 smithi160 ceph-mon[36639]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:44.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:44 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/1107187537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da4423f0-17d2-4449-b054-c46711cc0786"}]: dispatch 2024-06-01T00:50:44.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:44 smithi160 ceph-mon[36639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da4423f0-17d2-4449-b054-c46711cc0786"}]: dispatch 2024-06-01T00:50:44.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:44 smithi160 ceph-mon[36639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da4423f0-17d2-4449-b054-c46711cc0786"}]': finished 2024-06-01T00:50:44.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:44 smithi160 ceph-mon[36639]: osdmap e37: 6 total, 5 up, 6 in 2024-06-01T00:50:44.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:44 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:50:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:44 smithi079 ceph-mon[29356]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:44 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/1107187537' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da4423f0-17d2-4449-b054-c46711cc0786"}]: dispatch 2024-06-01T00:50:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:44 smithi079 ceph-mon[29356]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da4423f0-17d2-4449-b054-c46711cc0786"}]: dispatch 2024-06-01T00:50:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:44 smithi079 ceph-mon[29356]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da4423f0-17d2-4449-b054-c46711cc0786"}]': finished 2024-06-01T00:50:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:44 smithi079 ceph-mon[29356]: osdmap e37: 6 total, 5 up, 6 in 2024-06-01T00:50:44.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:44 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:50:45.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:45 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/287170464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:50:45.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:45 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/287170464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T00:50:46.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:46 smithi160 ceph-mon[36639]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:46.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:46 smithi079 ceph-mon[29356]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:48.497 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:48 smithi160 ceph-mon[36639]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:48.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:48 smithi079 ceph-mon[29356]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:50.498 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:50 smithi160 ceph-mon[36639]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:50.498 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:50 smithi079 ceph-mon[29356]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:50:51.475 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:51 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T00:50:51.475 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:51 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:51.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:51 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T00:50:51.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:51 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:50:52.574 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:52 smithi160 ceph-mon[36639]: Deploying daemon osd.5 on smithi160 2024-06-01T00:50:52.574 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:52 smithi160 ceph-mon[36639]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:52.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:52 smithi079 ceph-mon[29356]: Deploying daemon osd.5 on smithi160 2024-06-01T00:50:52.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:52 smithi079 ceph-mon[29356]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:54.656 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:54 smithi160 ceph-mon[36639]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:54.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:54 smithi079 ceph-mon[29356]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:56.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:56 smithi160 ceph-mon[36639]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:56.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:56.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:56.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:56 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:56 smithi079 ceph-mon[29356]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:50:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:56.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:56 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:50:58.639 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:50:58 smithi160 ceph-mon[36639]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:58.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:50:58 smithi079 ceph-mon[29356]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:50:59.825 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.823+0000 7fde9cff9640 1 -- 172.21.15.160:0/4248815034 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdea002c450 con 0x7fde8806af00 2024-06-01T00:50:59.826 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 5 on host 'smithi160' 2024-06-01T00:50:59.827 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.825+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde8806af00 msgr2=0x7fde8806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.825+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde8806af00 0x7fde8806d3c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdea01001e0 tx=0x7fde90008040 comp rx=0 tx=0).stop 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 msgr2=0x7fdea019ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea019ac50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fde9402fd20 tx=0x7fde9402f730 comp rx=0 tx=0).stop 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 shutdown_connections 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fde8806af00 0x7fde8806d3c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fdea0100380 0x7fdea019ac50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 --2- 172.21.15.160:0/4248815034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea00ff180 0x7fdea0069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 >> 172.21.15.160:0/4248815034 conn(0x7fdea00fa720 msgr2=0x7fdea0103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 shutdown_connections 2024-06-01T00:50:59.828 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:50:59.826+0000 7fdea5b0e640 1 -- 172.21.15.160:0/4248815034 wait complete. 2024-06-01T00:51:00.431 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:00 smithi160 ceph-mon[36639]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:51:00.431 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:00 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:00.431 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:00 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:00.432 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:00 smithi160 ceph-mon[36639]: from='osd.5 [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T00:51:00.432 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:00 smithi160 ceph-mon[36639]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T00:51:00.452 DEBUG:teuthology.orchestra.run.smithi160:osd.5> sudo journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.5.service 2024-06-01T00:51:00.454 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-01T00:51:00.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-01T00:51:00.674 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:00.796 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:00 smithi079 ceph-mon[29356]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:51:00.796 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:00 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:00.796 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:00 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:00.796 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:00 smithi079 ceph-mon[29356]: from='osd.5 [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T00:51:00.796 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:00 smithi079 ceph-mon[29356]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T00:51:01.917 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T00:51:01.917 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: osdmap e38: 6 total, 5 up, 6 in 2024-06-01T00:51:01.917 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:01.917 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: from='osd.5 [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:51:01.917 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:51:01.917 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:01.917 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:01.918 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:01 smithi160 ceph-mon[36639]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:51:02.069 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T00:51:02.069 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: osdmap e38: 6 total, 5 up, 6 in 2024-06-01T00:51:02.070 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:02.070 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: from='osd.5 [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:51:02.070 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-06-01T00:51:02.070 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:02.070 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:02.070 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:01 smithi079 ceph-mon[29356]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3577831892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 msgr2=0x7fb4b80ff5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3577831892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b80ff5c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8009880 tx=0x7fb4a802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3577831892 shutdown_connections 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3577831892 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4b81003a0 0x7fb4b8100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3577831892 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b80ff5c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3577831892 >> 172.21.15.79:0/3577831892 conn(0x7fb4b80fa760 msgr2=0x7fb4b80fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3577831892 shutdown_connections 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.406+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3577831892 wait complete. 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.407+0000 7fb4bdc78640 1 Processor -- start 2024-06-01T00:51:02.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.407+0000 7fb4bdc78640 1 -- start start 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.407+0000 7fb4bdc78640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b8103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.407+0000 7fb4bdc78640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4b81003a0 0x7fb4b8101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.407+0000 7fb4bdc78640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8102030 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.407+0000 7fb4bdc78640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b81021a0 con 0x7fb4b81003a0 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.407+0000 7fb4b77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b8103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b8103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50760/0 (socket says 172.21.15.79:50760) 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b77fe640 1 -- 172.21.15.79:0/3155690183 learned_addr learned my addr 172.21.15.79:0/3155690183 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b6ffd640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4b81003a0 0x7fb4b8101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b77fe640 1 -- 172.21.15.79:0/3155690183 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4b81003a0 msgr2=0x7fb4b8101af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b77fe640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4b81003a0 0x7fb4b8101af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b77fe640 1 -- 172.21.15.79:0/3155690183 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8009560 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.410 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b77fe640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b8103450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb4a802a9c0 tx=0x7fb4a80054f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b4ff9640 1 -- 172.21.15.79:0/3155690183 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a8004aa0 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b4ff9640 1 -- 172.21.15.79:0/3155690183 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4a8004c00 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b8102420 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4b4ff9640 1 -- 172.21.15.79:0/3155690183 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a8003a80 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.408+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b819dda0 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.409+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b804e210 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.410+0000 7fb4b4ff9640 1 -- 172.21.15.79:0/3155690183 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb4a8003d40 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.410+0000 7fb4b4ff9640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb4a006af00 0x7fb4a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.410+0000 7fb4b4ff9640 1 -- 172.21.15.79:0/3155690183 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4358+0+0 (secure 0 0 0) 0x7fb4a80aac80 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.410+0000 7fb4b6ffd640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb4a006af00 0x7fb4a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:02.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.410+0000 7fb4b6ffd640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb4a006af00 0x7fb4a006d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4b81031d0 tx=0x7fb4ac00b390 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:02.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.415+0000 7fb4b4ff9640 1 -- 172.21.15.79:0/3155690183 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb4a8076420 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.585+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb4b8107da0 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.587 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.586+0000 7fb4b4ff9640 1 -- 172.21.15.79:0/3155690183 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fb4b8107da0 con 0x7fb4b80ff1a0 2024-06-01T00:51:02.587 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb4a006af00 msgr2=0x7fb4a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb4a006af00 0x7fb4a006d3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4b81031d0 tx=0x7fb4ac00b390 comp rx=0 tx=0).stop 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 msgr2=0x7fb4b8103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b8103450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb4a802a9c0 tx=0x7fb4a80054f0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 shutdown_connections 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb4a006af00 0x7fb4a006d3c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb4b81003a0 0x7fb4b8101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.589 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 --2- 172.21.15.79:0/3155690183 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb4b80ff1a0 0x7fb4b8103450 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:02.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 >> 172.21.15.79:0/3155690183 conn(0x7fb4b80fa760 msgr2=0x7fb4b80687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:02.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 shutdown_connections 2024-06-01T00:51:02.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:02.588+0000 7fb4bdc78640 1 -- 172.21.15.79:0/3155690183 wait complete. 2024-06-01T00:51:02.804 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:51:02.637+0000 7f66728ef640 -1 osd.5 0 waiting for initial osdmap 2024-06-01T00:51:02.805 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:51:02.647+0000 7f666af22640 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T00:51:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:02 smithi079 ceph-mon[29356]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-06-01T00:51:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:02 smithi079 ceph-mon[29356]: osdmap e39: 6 total, 5 up, 6 in 2024-06-01T00:51:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:02 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:02 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:02 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3155690183' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-01T00:51:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:02 smithi079 ceph-mon[29356]: from='osd.5 ' entity='osd.5' 2024-06-01T00:51:03.086 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-mon[36639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-06-01T00:51:03.086 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-mon[36639]: osdmap e39: 6 total, 5 up, 6 in 2024-06-01T00:51:03.086 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:03.086 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:03.086 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3155690183' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-01T00:51:03.087 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:02 smithi160 ceph-mon[36639]: from='osd.5 ' entity='osd.5' 2024-06-01T00:51:03.287 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1717203037,"num_in_osds":6,"osd_in_since":1717203044,"num_remapped_pgs":0} 2024-06-01T00:51:04.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:03 smithi160 ceph-mon[36639]: purged_snaps scrub starts 2024-06-01T00:51:04.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:03 smithi160 ceph-mon[36639]: purged_snaps scrub ok 2024-06-01T00:51:04.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:03 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:04.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:03 smithi160 ceph-mon[36639]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:51:04.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:03 smithi160 ceph-mon[36639]: osd.5 [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] boot 2024-06-01T00:51:04.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:03 smithi160 ceph-mon[36639]: osdmap e40: 6 total, 6 up, 6 in 2024-06-01T00:51:04.031 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:03 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:04.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:03 smithi079 ceph-mon[29356]: purged_snaps scrub starts 2024-06-01T00:51:04.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:03 smithi079 ceph-mon[29356]: purged_snaps scrub ok 2024-06-01T00:51:04.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:03 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:04.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:03 smithi079 ceph-mon[29356]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-06-01T00:51:04.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:03 smithi079 ceph-mon[29356]: osd.5 [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] boot 2024-06-01T00:51:04.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:03 smithi079 ceph-mon[29356]: osdmap e40: 6 total, 6 up, 6 in 2024-06-01T00:51:04.106 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:03 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T00:51:04.289 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-01T00:51:04.511 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:05.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: Detected new or changed devices on smithi160 2024-06-01T00:51:05.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:51:05.681 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: Adjusting osd_memory_target on smithi160 to 4970M 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: osdmap e41: 6 total, 6 up, 6 in 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:51:05.682 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.683 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: Detected new or changed devices on smithi160 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: Adjusting osd_memory_target on smithi160 to 4970M 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:51:05.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:51:05.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: osdmap e41: 6 total, 6 up, 6 in 2024-06-01T00:51:05.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:51:05.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:51:05.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:51:05.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:06.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 -- 172.21.15.79:0/4271427058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a54100d70 msgr2=0x7f7a54103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 --2- 172.21.15.79:0/4271427058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a54100d70 0x7f7a54103160 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009880 tx=0x7f7a4402a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 -- 172.21.15.79:0/4271427058 shutdown_connections 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 --2- 172.21.15.79:0/4271427058 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a541036a0 0x7f7a54105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 --2- 172.21.15.79:0/4271427058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a54100d70 0x7f7a54103160 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 -- 172.21.15.79:0/4271427058 >> 172.21.15.79:0/4271427058 conn(0x7f7a540fa700 msgr2=0x7f7a540fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 -- 172.21.15.79:0/4271427058 shutdown_connections 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 -- 172.21.15.79:0/4271427058 wait complete. 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.239+0000 7f7a5951d640 1 Processor -- start 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a5951d640 1 -- start start 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a5951d640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a54100d70 0x7f7a54100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a5951d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a541036a0 0x7f7a540fed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:06.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a5951d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54100c00 con 0x7f7a541036a0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a5951d640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a540ff260 con 0x7f7a54100d70 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a4a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a541036a0 0x7f7a540fed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a4a7fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a541036a0 0x7f7a540fed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50776/0 (socket says 172.21.15.79:50776) 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a4a7fc640 1 -- 172.21.15.79:0/795109238 learned_addr learned my addr 172.21.15.79:0/795109238 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a52ffd640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a54100d70 0x7f7a54100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a4a7fc640 1 -- 172.21.15.79:0/795109238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a54100d70 msgr2=0x7f7a54100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a4a7fc640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a54100d70 0x7f7a54100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a4a7fc640 1 -- 172.21.15.79:0/795109238 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a44009560 con 0x7f7a541036a0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a4a7fc640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a541036a0 0x7f7a540fed20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00b3d0 tx=0x7f7a3c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.240+0000 7f7a50ff9640 1 -- 172.21.15.79:0/795109238 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a3c011800 con 0x7f7a541036a0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.241+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a540ff540 con 0x7f7a541036a0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.241+0000 7f7a50ff9640 1 -- 172.21.15.79:0/795109238 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a3c011e20 con 0x7f7a541036a0 2024-06-01T00:51:06.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.241+0000 7f7a50ff9640 1 -- 172.21.15.79:0/795109238 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a3c00f2e0 con 0x7f7a541036a0 2024-06-01T00:51:06.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.241+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a540ffb10 con 0x7f7a541036a0 2024-06-01T00:51:06.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.241+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a54108ef0 con 0x7f7a541036a0 2024-06-01T00:51:06.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.242+0000 7f7a50ff9640 1 -- 172.21.15.79:0/795109238 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7a3c010380 con 0x7f7a541036a0 2024-06-01T00:51:06.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.242+0000 7f7a50ff9640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7a4006aef0 0x7f7a4006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:06.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.243+0000 7f7a50ff9640 1 -- 172.21.15.79:0/795109238 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7a3c0897b0 con 0x7f7a541036a0 2024-06-01T00:51:06.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.243+0000 7f7a52ffd640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7a4006aef0 0x7f7a4006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:06.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.243+0000 7f7a52ffd640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7a4006aef0 0x7f7a4006d3b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000c00 tx=0x7f7a440314e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:06.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.247+0000 7f7a50ff9640 1 -- 172.21.15.79:0/795109238 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7a3c054e10 con 0x7f7a541036a0 2024-06-01T00:51:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.421+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7a5419e1d0 con 0x7f7a541036a0 2024-06-01T00:51:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.421+0000 7f7a50ff9640 1 -- 172.21.15.79:0/795109238 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f7a3c0150a0 con 0x7f7a541036a0 2024-06-01T00:51:06.422 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:06.424 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7a4006aef0 msgr2=0x7f7a4006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7a4006aef0 0x7f7a4006d3b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000c00 tx=0x7f7a440314e0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a541036a0 msgr2=0x7f7a540fed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a541036a0 0x7f7a540fed20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00b3d0 tx=0x7f7a3c00b890 comp rx=0 tx=0).stop 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 shutdown_connections 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7a4006aef0 0x7f7a4006d3b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7a541036a0 0x7f7a540fed20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 --2- 172.21.15.79:0/795109238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7a54100d70 0x7f7a54100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 >> 172.21.15.79:0/795109238 conn(0x7f7a540fa700 msgr2=0x7f7a540fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 shutdown_connections 2024-06-01T00:51:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:06.423+0000 7f7a5951d640 1 -- 172.21.15.79:0/795109238 wait complete. 2024-06-01T00:51:06.507 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:06 smithi079 ceph-mon[29356]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:06 smithi160 ceph-mon[36639]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:07.026 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1717203063,"num_in_osds":6,"osd_in_since":1717203044,"num_remapped_pgs":0} 2024-06-01T00:51:07.027 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-01T00:51:07.231 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:07.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:07 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/795109238' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-01T00:51:07.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:07 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/795109238' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-01T00:51:08.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:08 smithi079 ceph-mon[29356]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:08.706 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:08 smithi160 ceph-mon[36639]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:08.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 -- 172.21.15.79:0/1931995238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 msgr2=0x7fe84c10a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:08.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 --2- 172.21.15.79:0/1931995238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c10a1d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe840009880 tx=0x7fe84002a530 comp rx=0 tx=0).stop 2024-06-01T00:51:08.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 -- 172.21.15.79:0/1931995238 shutdown_connections 2024-06-01T00:51:08.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 --2- 172.21.15.79:0/1931995238 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c10a1d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:08.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 --2- 172.21.15.79:0/1931995238 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe84c1013c0 0x7fe84c1017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:08.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 -- 172.21.15.79:0/1931995238 >> 172.21.15.79:0/1931995238 conn(0x7fe84c075e10 msgr2=0x7fe84c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:08.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 -- 172.21.15.79:0/1931995238 shutdown_connections 2024-06-01T00:51:08.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 -- 172.21.15.79:0/1931995238 wait complete. 2024-06-01T00:51:08.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 Processor -- start 2024-06-01T00:51:08.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.744+0000 7fe8541b0640 1 -- start start 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe8541b0640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe84c1013c0 0x7fe84c196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe8541b0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c1967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe8541b0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c196e50 con 0x7fe84c101ce0 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe8541b0640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c19ab70 con 0x7fe84c1013c0 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe85174b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c1967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe85174b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c1967c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50784/0 (socket says 172.21.15.79:50784) 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe85174b640 1 -- 172.21.15.79:0/3671736179 learned_addr learned my addr 172.21.15.79:0/3671736179 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe851f4c640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe84c1013c0 0x7fe84c196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:08.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe85174b640 1 -- 172.21.15.79:0/3671736179 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe84c1013c0 msgr2=0x7fe84c196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe85174b640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe84c1013c0 0x7fe84c196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe85174b640 1 -- 172.21.15.79:0/3671736179 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840009560 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe85174b640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c1967c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe84002aa50 tx=0x7fe8400055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe84affd640 1 -- 172.21.15.79:0/3671736179 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe840004aa0 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c19adf0 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe84affd640 1 -- 172.21.15.79:0/3671736179 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe840004c00 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe84affd640 1 -- 172.21.15.79:0/3671736179 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8400039e0 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.745+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c19b2e0 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.746+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c068510 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.747+0000 7fe84affd640 1 -- 172.21.15.79:0/3671736179 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe84002ad50 con 0x7fe84c101ce0 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.747+0000 7fe84affd640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe83406aef0 0x7fe83406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:08.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.747+0000 7fe84affd640 1 -- 172.21.15.79:0/3671736179 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe8400aaa20 con 0x7fe84c101ce0 2024-06-01T00:51:08.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.747+0000 7fe851f4c640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe83406aef0 0x7fe83406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:08.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.748+0000 7fe851f4c640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe83406aef0 0x7fe83406d3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe83c005c40 tx=0x7fe83c008230 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:08.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.751+0000 7fe84affd640 1 -- 172.21.15.79:0/3671736179 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe840076000 con 0x7fe84c101ce0 2024-06-01T00:51:08.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.922+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe84c04e210 con 0x7fe84c101ce0 2024-06-01T00:51:08.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.922+0000 7fe84affd640 1 -- 172.21.15.79:0/3671736179 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11311 (secure 0 0 0) 0x7fe840079620 con 0x7fe84c101ce0 2024-06-01T00:51:08.924 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:08.924 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":41,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","created":"2024-06-01T00:45:32.072846+0000","modified":"2024-06-01T00:51:04.835598+0000","last_up_change":"2024-06-01T00:51:03.641391+0000","last_in_change":"2024-06-01T00:50:44.074666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-01T00:49:51.569670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a46a931e-6c2b-457c-808f-90274056b4f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6803","nonce":2649674789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6805","nonce":2649674789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6809","nonce":2649674789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6807","nonce":2649674789}]},"public_addr":"172.21.15.79:6803/2649674789","cluster_addr":"172.21.15.79:6805/2649674789","heartbeat_back_addr":"172.21.15.79:6809/2649674789","heartbeat_front_addr":"172.21.15.79:6807/2649674789","state":["exists","up"]},{"osd":1,"uuid":"ba57b192-9f40-405c-b51e-2ce8229d54d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6811","nonce":2565529362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6813","nonce":2565529362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6817","nonce":2565529362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6815","nonce":2565529362}]},"public_addr":"172.21.15.79:6811/2565529362","cluster_addr":"172.21.15.79:6813/2565529362","heartbeat_back_addr":"172.21.15.79:6817/2565529362","heartbeat_front_addr":"172.21.15.79:6815/2565529362","state":["exists","up"]},{"osd":2,"uuid":"9d75526b-189e-4b38-a3d2-8c26d77c70a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6819","nonce":2237394964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6821","nonce":2237394964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6825","nonce":2237394964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6823","nonce":2237394964}]},"public_addr":"172.21.15.79:6819/2237394964","cluster_addr":"172.21.15.79:6821/2237394964","heartbeat_back_addr":"172.21.15.79:6825/2237394964","heartbeat_front_addr":"172.21.15.79:6823/2237394964","state":["exists","up"]},{"osd":3,"uuid":"6b28a6b8-616c-4949-9ef8-8197138f624f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6801","nonce":70125620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6803","nonce":70125620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6807","nonce":70125620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6805","nonce":70125620}]},"public_addr":"172.21.15.160:6801/70125620","cluster_addr":"172.21.15.160:6803/70125620","heartbeat_back_addr":"172.21.15.160:6807/70125620","heartbeat_front_addr":"172.21.15.160:6805/70125620","state":["exists","up"]},{"osd":4,"uuid":"7adb728d-8dc0-4629-a8ef-03def9adf728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6809","nonce":3806998400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6811","nonce":3806998400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6815","nonce":3806998400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6813","nonce":3806998400}]},"public_addr":"172.21.15.160:6809/3806998400","cluster_addr":"172.21.15.160:6811/3806998400","heartbeat_back_addr":"172.21.15.160:6815/3806998400","heartbeat_front_addr":"172.21.15.160:6813/3806998400","state":["exists","up"]},{"osd":5,"uuid":"da4423f0-17d2-4449-b054-c46711cc0786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6817","nonce":610560416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6819","nonce":610560416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6823","nonce":610560416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6821","nonce":610560416}]},"public_addr":"172.21.15.160:6817/610560416","cluster_addr":"172.21.15.160:6819/610560416","heartbeat_back_addr":"172.21.15.160:6823/610560416","heartbeat_front_addr":"172.21.15.160:6821/610560416","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:48:50.800804+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:49:19.715204+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:49:47.430320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:50:10.851994+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:50:35.913170+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:51:01.266185+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1815731501":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3424688232":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3930502773":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/1853708063":"2024-06-02T00:46:27.100908+0000","172.21.15.79:0/3808174981":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6800/1904937508":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6801/616716500":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3503804565":"2024-06-02T00:46:27.100908+0000","172.21.15.79:0/51499797":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6800/616716500":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3979551222":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6801/1904937508":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6800/3983389754":"2024-06-02T00:47:35.256206+0000","172.21.15.79:0/1696807089":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6801/3983389754":"2024-06-02T00:47:35.256206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe83406aef0 msgr2=0x7fe83406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe83406aef0 0x7fe83406d3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe83c005c40 tx=0x7fe83c008230 comp rx=0 tx=0).stop 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 msgr2=0x7fe84c1967c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c1967c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe84002aa50 tx=0x7fe8400055a0 comp rx=0 tx=0).stop 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 shutdown_connections 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe83406aef0 0x7fe83406d3b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe84c101ce0 0x7fe84c1967c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 --2- 172.21.15.79:0/3671736179 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe84c1013c0 0x7fe84c196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.924+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 >> 172.21.15.79:0/3671736179 conn(0x7fe84c075e10 msgr2=0x7fe84c102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.925+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 shutdown_connections 2024-06-01T00:51:08.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:08.925+0000 7fe8541b0640 1 -- 172.21.15.79:0/3671736179 wait complete. 2024-06-01T00:51:09.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:09 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3671736179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T00:51:09.652 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-01T00:49:51.569670+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-01T00:51:09.652 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-01T00:51:09.727 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:09 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3671736179' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T00:51:09.856 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:10.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:10 smithi079 ceph-mon[29356]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:10.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:10 smithi160 ceph-mon[36639]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:11.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.436+0000 7fb2b509f640 1 -- 172.21.15.79:0/3416854193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b0101ce0 msgr2=0x7fb2b010a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:11.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.436+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3416854193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b0101ce0 0x7fb2b010a1d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb298009880 tx=0x7fb29802a530 comp rx=0 tx=0).stop 2024-06-01T00:51:11.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.436+0000 7fb2b509f640 1 -- 172.21.15.79:0/3416854193 shutdown_connections 2024-06-01T00:51:11.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.436+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3416854193 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b0101ce0 0x7fb2b010a1d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:11.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.436+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3416854193 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2b01013c0 0x7fb2b01017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.436+0000 7fb2b509f640 1 -- 172.21.15.79:0/3416854193 >> 172.21.15.79:0/3416854193 conn(0x7fb2b0075e10 msgr2=0x7fb2b00fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.436+0000 7fb2b509f640 1 -- 172.21.15.79:0/3416854193 shutdown_connections 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.437+0000 7fb2b509f640 1 -- 172.21.15.79:0/3416854193 wait complete. 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.437+0000 7fb2b509f640 1 Processor -- start 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.437+0000 7fb2b509f640 1 -- start start 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2b509f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b01013c0 0x7fb2b01963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2b509f640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2b0101ce0 0x7fb2b0196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2b509f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b0196f90 con 0x7fb2b01013c0 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2b509f640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b019acb0 con 0x7fb2b0101ce0 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2aed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b01013c0 0x7fb2b01963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2aed9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b01013c0 0x7fb2b01963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50806/0 (socket says 172.21.15.79:50806) 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2aed9d640 1 -- 172.21.15.79:0/3997719725 learned_addr learned my addr 172.21.15.79:0/3997719725 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:11.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2ae59c640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2b0101ce0 0x7fb2b0196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2aed9d640 1 -- 172.21.15.79:0/3997719725 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2b0101ce0 msgr2=0x7fb2b0196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2aed9d640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2b0101ce0 0x7fb2b0196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2aed9d640 1 -- 172.21.15.79:0/3997719725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb298009560 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2ae59c640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2b0101ce0 0x7fb2b0196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2aed9d640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b01013c0 0x7fb2b01963c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000ba50 tx=0x7fb2a000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2a7fff640 1 -- 172.21.15.79:0/3997719725 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2a000cb30 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b019af90 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2a7fff640 1 -- 172.21.15.79:0/3997719725 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb2a000cc90 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2a7fff640 1 -- 172.21.15.79:0/3997719725 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb2a0007740 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.438+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b019b4b0 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.440+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b0068510 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.440+0000 7fb2a7fff640 1 -- 172.21.15.79:0/3997719725 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb2a00079b0 con 0x7fb2b01013c0 2024-06-01T00:51:11.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.440+0000 7fb2a7fff640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb29c06af00 0x7fb29c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:11.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.440+0000 7fb2a7fff640 1 -- 172.21.15.79:0/3997719725 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb2a00895d0 con 0x7fb2b01013c0 2024-06-01T00:51:11.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.440+0000 7fb2ae59c640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb29c06af00 0x7fb29c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:11.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.441+0000 7fb2ae59c640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb29c06af00 0x7fb29c06d3c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb298009230 tx=0x7fb298005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:11.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.445+0000 7fb2a7fff640 1 -- 172.21.15.79:0/3997719725 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb2a0054c60 con 0x7fb2b01013c0 2024-06-01T00:51:11.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.618+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2b01976d0 con 0x7fb2b01013c0 2024-06-01T00:51:11.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.619+0000 7fb2a7fff640 1 -- 172.21.15.79:0/3997719725 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2a0058280 con 0x7fb2b01013c0 2024-06-01T00:51:11.620 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-06-01T00:51:11.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb29c06af00 msgr2=0x7fb29c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:11.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb29c06af00 0x7fb29c06d3c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb298009230 tx=0x7fb298005a20 comp rx=0 tx=0).stop 2024-06-01T00:51:11.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b01013c0 msgr2=0x7fb2b01963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:11.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b01013c0 0x7fb2b01963c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb2a000ba50 tx=0x7fb2a000bf10 comp rx=0 tx=0).stop 2024-06-01T00:51:11.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 shutdown_connections 2024-06-01T00:51:11.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb29c06af00 0x7fb29c06d3c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:11.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb2b0101ce0 0x7fb2b0196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:11.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 --2- 172.21.15.79:0/3997719725 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb2b01013c0 0x7fb2b01963c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:11.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 >> 172.21.15.79:0/3997719725 conn(0x7fb2b0075e10 msgr2=0x7fb2b01026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:11.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 shutdown_connections 2024-06-01T00:51:11.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:11.621+0000 7fb2b509f640 1 -- 172.21.15.79:0/3997719725 wait complete. 2024-06-01T00:51:12.171 INFO:tasks.cephadm:Setting up client nodes... 2024-06-01T00:51:12.171 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-01T00:51:12.374 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:12.702 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:12 smithi079 ceph-mon[29356]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:12.702 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:12 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3997719725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-01T00:51:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:12 smithi160 ceph-mon[36639]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:12 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3997719725' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 -- 172.21.15.79:0/914883463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 msgr2=0x7f143c100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 --2- 172.21.15.79:0/914883463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c100820 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1430009880 tx=0x7f143002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 -- 172.21.15.79:0/914883463 shutdown_connections 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 --2- 172.21.15.79:0/914883463 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c100820 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 --2- 172.21.15.79:0/914883463 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f143c0ff1a0 0x7f143c0ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 -- 172.21.15.79:0/914883463 >> 172.21.15.79:0/914883463 conn(0x7f143c0fa760 msgr2=0x7f143c0fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 -- 172.21.15.79:0/914883463 shutdown_connections 2024-06-01T00:51:14.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.008+0000 7f1441c56640 1 -- 172.21.15.79:0/914883463 wait complete. 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f1441c56640 1 Processor -- start 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f1441c56640 1 -- start start 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f1441c56640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f143c0ff1a0 0x7f143c195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f1441c56640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f1441c56640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c196a20 con 0x7f143c1003a0 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f1441c56640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c196b90 con 0x7f143c0ff1a0 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f143affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f143affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53662/0 (socket says 172.21.15.79:53662) 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f143affd640 1 -- 172.21.15.79:0/3310959174 learned_addr learned my addr 172.21.15.79:0/3310959174 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.009+0000 7f143b7fe640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f143c0ff1a0 0x7f143c195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f143affd640 1 -- 172.21.15.79:0/3310959174 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f143c0ff1a0 msgr2=0x7f143c195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f143affd640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f143c0ff1a0 0x7f143c195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:14.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f143affd640 1 -- 172.21.15.79:0/3310959174 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1430009560 con 0x7f143c1003a0 2024-06-01T00:51:14.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f143affd640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c196450 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1430005640 tx=0x7f1430005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:14.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f1438ff9640 1 -- 172.21.15.79:0/3310959174 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1430004ba0 con 0x7f143c1003a0 2024-06-01T00:51:14.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f1438ff9640 1 -- 172.21.15.79:0/3310959174 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1430004d00 con 0x7f143c1003a0 2024-06-01T00:51:14.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143c19b530 con 0x7f143c1003a0 2024-06-01T00:51:14.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f1438ff9640 1 -- 172.21.15.79:0/3310959174 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1430003c30 con 0x7f143c1003a0 2024-06-01T00:51:14.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.010+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143c19baa0 con 0x7f143c1003a0 2024-06-01T00:51:14.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.011+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c107e20 con 0x7f143c1003a0 2024-06-01T00:51:14.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.011+0000 7f1438ff9640 1 -- 172.21.15.79:0/3310959174 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f1430031640 con 0x7f143c1003a0 2024-06-01T00:51:14.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.012+0000 7f1438ff9640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f142406aef0 0x7f142406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:14.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.012+0000 7f143b7fe640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f142406aef0 0x7f142406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:14.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.012+0000 7f1438ff9640 1 -- 172.21.15.79:0/3310959174 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f14300aaf10 con 0x7f143c1003a0 2024-06-01T00:51:14.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.012+0000 7f143b7fe640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f142406aef0 0x7f142406d3b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f143c100200 tx=0x7f142c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:14.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.016+0000 7f1438ff9640 1 -- 172.21.15.79:0/3310959174 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f14300764f0 con 0x7f143c1003a0 2024-06-01T00:51:14.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.254+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f143c108010 con 0x7f143c1003a0 2024-06-01T00:51:14.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.257+0000 7f1438ff9640 1 -- 172.21.15.79:0/3310959174 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1430079b10 con 0x7f143c1003a0 2024-06-01T00:51:14.258 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-06-01T00:51:14.259 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQCCcFpmnF04DxAA2/mmJQPAGVqVmvxBrlfiMQ== 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.259+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f142406aef0 msgr2=0x7f142406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.259+0000 7f1441c56640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f142406aef0 0x7f142406d3b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f143c100200 tx=0x7f142c008040 comp rx=0 tx=0).stop 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.259+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 msgr2=0x7f143c196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.259+0000 7f1441c56640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c196450 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1430005640 tx=0x7f1430005670 comp rx=0 tx=0).stop 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.260+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 shutdown_connections 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.260+0000 7f1441c56640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f142406aef0 0x7f142406d3b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.260+0000 7f1441c56640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f143c1003a0 0x7f143c196450 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:14.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.260+0000 7f1441c56640 1 --2- 172.21.15.79:0/3310959174 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f143c0ff1a0 0x7f143c195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:14.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.260+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 >> 172.21.15.79:0/3310959174 conn(0x7f143c0fa760 msgr2=0x7f143c0fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:14.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.260+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 shutdown_connections 2024-06-01T00:51:14.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:14.260+0000 7f1441c56640 1 -- 172.21.15.79:0/3310959174 wait complete. 2024-06-01T00:51:14.607 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:14 smithi079 ceph-mon[29356]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:14.608 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:14 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3310959174' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T00:51:14.608 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:14 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3310959174' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T00:51:14.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:14 smithi160 ceph-mon[36639]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:14.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:14 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3310959174' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T00:51:14.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:14 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3310959174' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T00:51:15.002 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-06-01T00:51:15.002 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-01T00:51:15.002 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-01T00:51:15.039 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-01T00:51:15.257 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi160/config 2024-06-01T00:51:16.654 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:16 smithi160 ceph-mon[36639]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:16.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:16 smithi079 ceph-mon[29356]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:16.864 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 -- 172.21.15.160:0/229762043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c0ff180 msgr2=0x7f7b2c0ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:16.864 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/229762043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c0ff180 0x7f7b2c0ff5a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009880 tx=0x7f7b2002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:16.864 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 -- 172.21.15.160:0/229762043 shutdown_connections 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/229762043 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c100380 0x7f7b2c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/229762043 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c0ff180 0x7f7b2c0ff5a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 -- 172.21.15.160:0/229762043 >> 172.21.15.160:0/229762043 conn(0x7f7b2c0fa760 msgr2=0x7f7b2c0fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 -- 172.21.15.160:0/229762043 shutdown_connections 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 -- 172.21.15.160:0/229762043 wait complete. 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 Processor -- start 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.862+0000 7f7b32ae2640 1 -- start start 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b32ae2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0ff180 0x7f7b2c195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b32ae2640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c100380 0x7f7b2c1962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b32ae2640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c1968c0 con 0x7f7b2c0ff180 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b32ae2640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c196a30 con 0x7f7b2c100380 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b2bfff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c100380 0x7f7b2c1962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:16.865 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b2bfff640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c100380 0x7f7b2c1962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.160:53020/0 (socket says 172.21.15.160:53020) 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b2bfff640 1 -- 172.21.15.160:0/2089825566 learned_addr learned my addr 172.21.15.160:0/2089825566 (peer_addr_for_me v2:172.21.15.160:0/0) 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b3087e640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0ff180 0x7f7b2c195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b2bfff640 1 -- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0ff180 msgr2=0x7f7b2c195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b2bfff640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0ff180 0x7f7b2c195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b2bfff640 1 -- 172.21.15.160:0/2089825566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20009560 con 0x7f7b2c100380 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b2bfff640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c100380 0x7f7b2c1962f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c8f0 tx=0x7f7b1800cdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b29ffb640 1 -- 172.21.15.160:0/2089825566 <== mon.1 v2:172.21.15.160:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b18007b80 con 0x7f7b2c100380 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b29ffb640 1 -- 172.21.15.160:0/2089825566 <== mon.1 v2:172.21.15.160:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b180054e0 con 0x7f7b2c100380 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c19b480 con 0x7f7b2c100380 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.863+0000 7f7b29ffb640 1 -- 172.21.15.160:0/2089825566 <== mon.1 v2:172.21.15.160:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b1800f450 con 0x7f7b2c100380 2024-06-01T00:51:16.866 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.864+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c19ba50 con 0x7f7b2c100380 2024-06-01T00:51:16.867 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.865+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c107d80 con 0x7f7b2c100380 2024-06-01T00:51:16.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.865+0000 7f7b29ffb640 1 -- 172.21.15.160:0/2089825566 <== mon.1 v2:172.21.15.160:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7b180046c0 con 0x7f7b2c100380 2024-06-01T00:51:16.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.866+0000 7f7b29ffb640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b1c06aef0 0x7f7b1c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:16.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.866+0000 7f7b29ffb640 1 -- 172.21.15.160:0/2089825566 <== mon.1 v2:172.21.15.160:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7b18089940 con 0x7f7b2c100380 2024-06-01T00:51:16.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.866+0000 7f7b3087e640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b1c06aef0 0x7f7b1c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:16.868 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.866+0000 7f7b3087e640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b1c06aef0 0x7f7b1c06d3b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009230 tx=0x7f7b200314e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:16.870 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:16.870+0000 7f7b29ffb640 1 -- 172.21.15.160:0/2089825566 <== mon.1 v2:172.21.15.160:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7b18054fd0 con 0x7f7b2c100380 2024-06-01T00:51:17.112 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.112+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f7b2c19bcd0 con 0x7f7b2c100380 2024-06-01T00:51:17.117 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.117+0000 7f7b29ffb640 1 -- 172.21.15.160:0/2089825566 <== mon.1 v2:172.21.15.160:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f7b2c19bcd0 con 0x7f7b2c100380 2024-06-01T00:51:17.118 INFO:teuthology.orchestra.run.smithi160.stdout:[client.1] 2024-06-01T00:51:17.118 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCFcFpmPdTFBhAAEszFqZvAzs0qxViyHr6bKQ== 2024-06-01T00:51:17.120 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.119+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b1c06aef0 msgr2=0x7f7b1c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:17.120 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.119+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b1c06aef0 0x7f7b1c06d3b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7b20009230 tx=0x7f7b200314e0 comp rx=0 tx=0).stop 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.119+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c100380 msgr2=0x7f7b2c1962f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.119+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c100380 0x7f7b2c1962f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c8f0 tx=0x7f7b1800cdb0 comp rx=0 tx=0).stop 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.120+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 shutdown_connections 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.120+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b1c06aef0 0x7f7b1c06d3b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.120+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b2c100380 0x7f7b2c1962f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.120+0000 7f7b32ae2640 1 --2- 172.21.15.160:0/2089825566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b2c0ff180 0x7f7b2c195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.120+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 >> 172.21.15.160:0/2089825566 conn(0x7f7b2c0fa760 msgr2=0x7f7b2c0687d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.120+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 shutdown_connections 2024-06-01T00:51:17.121 INFO:teuthology.orchestra.run.smithi160.stderr:2024-06-01T00:51:17.120+0000 7f7b32ae2640 1 -- 172.21.15.160:0/2089825566 wait complete. 2024-06-01T00:51:17.703 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:17 smithi160 ceph-mon[36639]: from='client.? 172.21.15.160:0/2089825566' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T00:51:17.703 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:17 smithi160 ceph-mon[36639]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T00:51:17.703 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:17 smithi160 ceph-mon[36639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T00:51:17.724 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-06-01T00:51:17.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-01T00:51:17.724 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-01T00:51:17.762 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-01T00:51:17.762 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-01T00:51:17.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-01T00:51:17.788 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:17 smithi079 ceph-mon[29356]: from='client.? 172.21.15.160:0/2089825566' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T00:51:17.789 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:17 smithi079 ceph-mon[29356]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T00:51:17.789 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:17 smithi079 ceph-mon[29356]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T00:51:17.978 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:18 smithi160 ceph-mon[36639]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:18.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:18 smithi079 ceph-mon[29356]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:19.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 -- 172.21.15.79:0/903253012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101d30 msgr2=0x7f1e9c10a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:19.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 --2- 172.21.15.79:0/903253012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101d30 0x7f1e9c10a220 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009880 tx=0x7f1e8402a530 comp rx=0 tx=0).stop 2024-06-01T00:51:19.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 -- 172.21.15.79:0/903253012 shutdown_connections 2024-06-01T00:51:19.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 --2- 172.21.15.79:0/903253012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101d30 0x7f1e9c10a220 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 --2- 172.21.15.79:0/903253012 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1e9c101410 0x7f1e9c1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 -- 172.21.15.79:0/903253012 >> 172.21.15.79:0/903253012 conn(0x7f1e9c0fac80 msgr2=0x7f1e9c0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 -- 172.21.15.79:0/903253012 shutdown_connections 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.526+0000 7f1ea2963640 1 -- 172.21.15.79:0/903253012 wait complete. 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1ea2963640 1 Processor -- start 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1ea2963640 1 -- start start 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1ea2963640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101410 0x7f1e9c196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1ea2963640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1e9c101d30 0x7f1e9c196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1ea2963640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c196f00 con 0x7f1e9c101410 2024-06-01T00:51:19.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1ea2963640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c19ac20 con 0x7f1e9c101d30 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e9bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101410 0x7f1e9c196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e9bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101410 0x7f1e9c196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53678/0 (socket says 172.21.15.79:53678) 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e9bfff640 1 -- 172.21.15.79:0/1384447437 learned_addr learned my addr 172.21.15.79:0/1384447437 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e93fff640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1e9c101d30 0x7f1e9c196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e9bfff640 1 -- 172.21.15.79:0/1384447437 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1e9c101d30 msgr2=0x7f1e9c196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e9bfff640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1e9c101d30 0x7f1e9c196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e9bfff640 1 -- 172.21.15.79:0/1384447437 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84009560 con 0x7f1e9c101410 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.527+0000 7f1e93fff640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1e9c101d30 0x7f1e9c196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.528+0000 7f1e9bfff640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101410 0x7f1e9c196330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00b420 tx=0x7f1e8c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:19.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.528+0000 7f1e99ffb640 1 -- 172.21.15.79:0/1384447437 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e8c011800 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.528+0000 7f1e99ffb640 1 -- 172.21.15.79:0/1384447437 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e8c011e20 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.528+0000 7f1e99ffb640 1 -- 172.21.15.79:0/1384447437 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e8c00f2e0 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.528+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9c19af00 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.528+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9c19b450 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.529+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9c069060 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.529+0000 7f1e99ffb640 1 -- 172.21.15.79:0/1384447437 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f1e8c010380 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.530+0000 7f1e99ffb640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1e8806aef0 0x7f1e8806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.530+0000 7f1e93fff640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1e8806aef0 0x7f1e8806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.530+0000 7f1e99ffb640 1 -- 172.21.15.79:0/1384447437 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1e8c0894d0 con 0x7f1e9c101410 2024-06-01T00:51:19.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.530+0000 7f1e93fff640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1e8806aef0 0x7f1e8806d3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1e84000c00 tx=0x7f1e84005e60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:19.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.534+0000 7f1e99ffb640 1 -- 172.21.15.79:0/1384447437 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1e8c054b30 con 0x7f1e9c101410 2024-06-01T00:51:19.755 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.754+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1e9c04e210 con 0x7f1e9c101410 2024-06-01T00:51:19.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.757+0000 7f1e99ffb640 1 -- 172.21.15.79:0/1384447437 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+171348 (secure 0 0 0) 0x7f1e8c058150 con 0x7f1e9c101410 2024-06-01T00:51:19.760 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:19.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1e8806aef0 msgr2=0x7f1e8806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:19.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1e8806aef0 0x7f1e8806d3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1e84000c00 tx=0x7f1e84005e60 comp rx=0 tx=0).stop 2024-06-01T00:51:19.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101410 msgr2=0x7f1e9c196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:19.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101410 0x7f1e9c196330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00b420 tx=0x7f1e8c00b8e0 comp rx=0 tx=0).stop 2024-06-01T00:51:19.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 shutdown_connections 2024-06-01T00:51:19.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1e8806aef0 0x7f1e8806d3b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:19.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1e9c101d30 0x7f1e9c196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:19.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 --2- 172.21.15.79:0/1384447437 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1e9c101410 0x7f1e9c196330 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:19.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 >> 172.21.15.79:0/1384447437 conn(0x7f1e9c0fac80 msgr2=0x7f1e9c102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:19.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 shutdown_connections 2024-06-01T00:51:19.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:19.759+0000 7f1ea2963640 1 -- 172.21.15.79:0/1384447437 wait complete. 2024-06-01T00:51:20.473 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":19,"flags":0,"active_gid":14215,"active_name":"smithi079.emylav","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":187237089},{"type":"v1","addr":"172.21.15.79:6801","nonce":187237089}]},"active_addr":"172.21.15.79:6801/187237089","active_change":"2024-06-01T00:47:35.256313+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14238,"name":"smithi160.dxcuke","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.79:8443/","prometheus":"http://172.21.15.79:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":12524768}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3484477061}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":2126066860}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3152154540}]}]} 2024-06-01T00:51:20.477 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-01T00:51:20.477 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-01T00:51:20.477 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-01T00:51:20.714 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:20 smithi160 ceph-mon[36639]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:20.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:20 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1384447437' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-01T00:51:20.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:20.812 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:20 smithi079 ceph-mon[29356]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:20.812 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:20 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1384447437' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-01T00:51:20.812 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 -- 172.21.15.79:0/4176211747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4438101ce0 msgr2=0x7f443810a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 --2- 172.21.15.79:0/4176211747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4438101ce0 0x7f443810a1d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f442c009880 tx=0x7f442c02a530 comp rx=0 tx=0).stop 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 -- 172.21.15.79:0/4176211747 shutdown_connections 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 --2- 172.21.15.79:0/4176211747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4438101ce0 0x7f443810a1d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 --2- 172.21.15.79:0/4176211747 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f44381013c0 0x7f44381017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 -- 172.21.15.79:0/4176211747 >> 172.21.15.79:0/4176211747 conn(0x7f4438075e10 msgr2=0x7f44380fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 -- 172.21.15.79:0/4176211747 shutdown_connections 2024-06-01T00:51:22.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.098+0000 7f443e045640 1 -- 172.21.15.79:0/4176211747 wait complete. 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.099+0000 7f443e045640 1 Processor -- start 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.099+0000 7f443e045640 1 -- start start 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.099+0000 7f443e045640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44381013c0 0x7f44381963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.099+0000 7f443e045640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4438101ce0 0x7f4438196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.099+0000 7f443e045640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4438196f90 con 0x7f44381013c0 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.099+0000 7f443e045640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443819acb0 con 0x7f4438101ce0 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f4436ffd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4438101ce0 0x7f4438196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f44377fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44381013c0 0x7f44381963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f4436ffd640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4438101ce0 0x7f4438196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.160:3300/0 says I am v2:172.21.15.79:46424/0 (socket says 172.21.15.79:46424) 2024-06-01T00:51:22.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f44377fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44381013c0 0x7f44381963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49144/0 (socket says 172.21.15.79:49144) 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f4436ffd640 1 -- 172.21.15.79:0/2818653381 learned_addr learned my addr 172.21.15.79:0/2818653381 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f44377fe640 1 -- 172.21.15.79:0/2818653381 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4438101ce0 msgr2=0x7f4438196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f44377fe640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4438101ce0 0x7f4438196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f44377fe640 1 -- 172.21.15.79:0/2818653381 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c009560 con 0x7f44381013c0 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f44377fe640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44381013c0 0x7f44381963c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f442800b370 tx=0x7f442800b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f4434ff9640 1 -- 172.21.15.79:0/2818653381 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4428011800 con 0x7f44381013c0 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443819af90 con 0x7f44381013c0 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f4434ff9640 1 -- 172.21.15.79:0/2818653381 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4428011e20 con 0x7f44381013c0 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f4434ff9640 1 -- 172.21.15.79:0/2818653381 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f442800f2e0 con 0x7f44381013c0 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.100+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443819b4e0 con 0x7f44381013c0 2024-06-01T00:51:22.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.101+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4438068510 con 0x7f44381013c0 2024-06-01T00:51:22.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.102+0000 7f4434ff9640 1 -- 172.21.15.79:0/2818653381 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f4428010380 con 0x7f44381013c0 2024-06-01T00:51:22.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.102+0000 7f4434ff9640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f442006aef0 0x7f442006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:22.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.102+0000 7f4434ff9640 1 -- 172.21.15.79:0/2818653381 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f44280897c0 con 0x7f44381013c0 2024-06-01T00:51:22.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.102+0000 7f4436ffd640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f442006aef0 0x7f442006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:22.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.103+0000 7f4436ffd640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f442006aef0 0x7f442006d3b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4438197920 tx=0x7f442c0058d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:22.124 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.106+0000 7f4434ff9640 1 -- 172.21.15.79:0/2818653381 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4428054da0 con 0x7f44381013c0 2024-06-01T00:51:22.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.280+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f443804e210 con 0x7f44381013c0 2024-06-01T00:51:22.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.281+0000 7f4434ff9640 1 -- 172.21.15.79:0/2818653381 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11311 (secure 0 0 0) 0x7f44280583c0 con 0x7f44381013c0 2024-06-01T00:51:22.282 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:22.282 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":41,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","created":"2024-06-01T00:45:32.072846+0000","modified":"2024-06-01T00:51:04.835598+0000","last_up_change":"2024-06-01T00:51:03.641391+0000","last_in_change":"2024-06-01T00:50:44.074666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-01T00:49:51.569670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a46a931e-6c2b-457c-808f-90274056b4f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6803","nonce":2649674789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6805","nonce":2649674789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6809","nonce":2649674789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6807","nonce":2649674789}]},"public_addr":"172.21.15.79:6803/2649674789","cluster_addr":"172.21.15.79:6805/2649674789","heartbeat_back_addr":"172.21.15.79:6809/2649674789","heartbeat_front_addr":"172.21.15.79:6807/2649674789","state":["exists","up"]},{"osd":1,"uuid":"ba57b192-9f40-405c-b51e-2ce8229d54d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6811","nonce":2565529362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6813","nonce":2565529362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6817","nonce":2565529362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6815","nonce":2565529362}]},"public_addr":"172.21.15.79:6811/2565529362","cluster_addr":"172.21.15.79:6813/2565529362","heartbeat_back_addr":"172.21.15.79:6817/2565529362","heartbeat_front_addr":"172.21.15.79:6815/2565529362","state":["exists","up"]},{"osd":2,"uuid":"9d75526b-189e-4b38-a3d2-8c26d77c70a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6819","nonce":2237394964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6821","nonce":2237394964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6825","nonce":2237394964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6823","nonce":2237394964}]},"public_addr":"172.21.15.79:6819/2237394964","cluster_addr":"172.21.15.79:6821/2237394964","heartbeat_back_addr":"172.21.15.79:6825/2237394964","heartbeat_front_addr":"172.21.15.79:6823/2237394964","state":["exists","up"]},{"osd":3,"uuid":"6b28a6b8-616c-4949-9ef8-8197138f624f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6801","nonce":70125620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6803","nonce":70125620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6807","nonce":70125620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6805","nonce":70125620}]},"public_addr":"172.21.15.160:6801/70125620","cluster_addr":"172.21.15.160:6803/70125620","heartbeat_back_addr":"172.21.15.160:6807/70125620","heartbeat_front_addr":"172.21.15.160:6805/70125620","state":["exists","up"]},{"osd":4,"uuid":"7adb728d-8dc0-4629-a8ef-03def9adf728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6809","nonce":3806998400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6811","nonce":3806998400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6815","nonce":3806998400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6813","nonce":3806998400}]},"public_addr":"172.21.15.160:6809/3806998400","cluster_addr":"172.21.15.160:6811/3806998400","heartbeat_back_addr":"172.21.15.160:6815/3806998400","heartbeat_front_addr":"172.21.15.160:6813/3806998400","state":["exists","up"]},{"osd":5,"uuid":"da4423f0-17d2-4449-b054-c46711cc0786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6817","nonce":610560416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6819","nonce":610560416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6823","nonce":610560416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6821","nonce":610560416}]},"public_addr":"172.21.15.160:6817/610560416","cluster_addr":"172.21.15.160:6819/610560416","heartbeat_back_addr":"172.21.15.160:6823/610560416","heartbeat_front_addr":"172.21.15.160:6821/610560416","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:48:50.800804+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:49:19.715204+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:49:47.430320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:50:10.851994+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:50:35.913170+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:51:01.266185+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1815731501":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3424688232":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3930502773":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/1853708063":"2024-06-02T00:46:27.100908+0000","172.21.15.79:0/3808174981":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6800/1904937508":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6801/616716500":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3503804565":"2024-06-02T00:46:27.100908+0000","172.21.15.79:0/51499797":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6800/616716500":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3979551222":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6801/1904937508":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6800/3983389754":"2024-06-02T00:47:35.256206+0000","172.21.15.79:0/1696807089":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6801/3983389754":"2024-06-02T00:47:35.256206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f442006aef0 msgr2=0x7f442006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f442006aef0 0x7f442006d3b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4438197920 tx=0x7f442c0058d0 comp rx=0 tx=0).stop 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44381013c0 msgr2=0x7f44381963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44381013c0 0x7f44381963c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f442800b370 tx=0x7f442800b830 comp rx=0 tx=0).stop 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 shutdown_connections 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f442006aef0 0x7f442006d3b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4438101ce0 0x7f4438196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 --2- 172.21.15.79:0/2818653381 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f44381013c0 0x7f44381963c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 >> 172.21.15.79:0/2818653381 conn(0x7f4438075e10 msgr2=0x7f44381026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:22.284 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 shutdown_connections 2024-06-01T00:51:22.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:22.283+0000 7f443e045640 1 -- 172.21.15.79:0/2818653381 wait complete. 2024-06-01T00:51:22.683 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:22 smithi079 ceph-mon[29356]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:22.683 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:22 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2818653381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T00:51:22.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:22 smithi160 ceph-mon[36639]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:22.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:22 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2818653381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T00:51:23.029 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-01T00:51:23.030 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-01T00:51:23.228 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:24.558 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:24 smithi079 ceph-mon[29356]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:24 smithi160 ceph-mon[36639]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:24.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.811+0000 7f7d3834a640 1 -- 172.21.15.79:0/1308570819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ff360 msgr2=0x7f7d300ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:24.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.811+0000 7f7d3834a640 1 --2- 172.21.15.79:0/1308570819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ff360 0x7f7d300ff740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009880 tx=0x7f7d2c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.811+0000 7f7d3834a640 1 -- 172.21.15.79:0/1308570819 shutdown_connections 2024-06-01T00:51:24.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.811+0000 7f7d3834a640 1 --2- 172.21.15.79:0/1308570819 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d300ffd10 0x7f7d30103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.811+0000 7f7d3834a640 1 --2- 172.21.15.79:0/1308570819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ff360 0x7f7d300ff740 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.811+0000 7f7d3834a640 1 -- 172.21.15.79:0/1308570819 >> 172.21.15.79:0/1308570819 conn(0x7f7d30075e10 msgr2=0x7f7d300fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:24.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 -- 172.21.15.79:0/1308570819 shutdown_connections 2024-06-01T00:51:24.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 -- 172.21.15.79:0/1308570819 wait complete. 2024-06-01T00:51:24.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 Processor -- start 2024-06-01T00:51:24.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 -- start start 2024-06-01T00:51:24.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d300ff360 0x7f7d301970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:24.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ffd10 0x7f7d301975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:24.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30197b30 con 0x7f7d300ffd10 2024-06-01T00:51:24.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d3834a640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30197ca0 con 0x7f7d300ff360 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d358e5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ffd10 0x7f7d301975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d358e5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ffd10 0x7f7d301975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49168/0 (socket says 172.21.15.79:49168) 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d358e5640 1 -- 172.21.15.79:0/3853236069 learned_addr learned my addr 172.21.15.79:0/3853236069 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.812+0000 7f7d360e6640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d300ff360 0x7f7d301970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d358e5640 1 -- 172.21.15.79:0/3853236069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d300ff360 msgr2=0x7f7d301970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d358e5640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d300ff360 0x7f7d301970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d358e5640 1 -- 172.21.15.79:0/3853236069 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c009560 con 0x7f7d300ffd10 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d358e5640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ffd10 0x7f7d301975f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d6c0 tx=0x7f7d2000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d277fe640 1 -- 172.21.15.79:0/3853236069 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d200051f0 con 0x7f7d300ffd10 2024-06-01T00:51:24.817 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d277fe640 1 -- 172.21.15.79:0/3853236069 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d20009ce0 con 0x7f7d300ffd10 2024-06-01T00:51:24.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d301a0120 con 0x7f7d300ffd10 2024-06-01T00:51:24.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d277fe640 1 -- 172.21.15.79:0/3853236069 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d20010460 con 0x7f7d300ffd10 2024-06-01T00:51:24.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.813+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d301a0530 con 0x7f7d300ffd10 2024-06-01T00:51:24.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.814+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d30100840 con 0x7f7d300ffd10 2024-06-01T00:51:24.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.815+0000 7f7d277fe640 1 -- 172.21.15.79:0/3853236069 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7d20005350 con 0x7f7d300ffd10 2024-06-01T00:51:24.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.815+0000 7f7d277fe640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7d1c06aef0 0x7f7d1c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:24.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.815+0000 7f7d360e6640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7d1c06aef0 0x7f7d1c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:24.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.815+0000 7f7d277fe640 1 -- 172.21.15.79:0/3853236069 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d20089510 con 0x7f7d300ffd10 2024-06-01T00:51:24.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.815+0000 7f7d360e6640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7d1c06aef0 0x7f7d1c06d3b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009230 tx=0x7f7d2c0058d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:24.821 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.819+0000 7f7d277fe640 1 -- 172.21.15.79:0/3853236069 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7d20054c20 con 0x7f7d300ffd10 2024-06-01T00:51:24.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.991+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d3010c3b0 con 0x7f7d300ffd10 2024-06-01T00:51:24.993 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.992+0000 7f7d277fe640 1 -- 172.21.15.79:0/3853236069 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11311 (secure 0 0 0) 0x7f7d20020090 con 0x7f7d300ffd10 2024-06-01T00:51:24.994 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:24.994 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":41,"fsid":"22870f5e-1fb0-11ef-bc9b-c7b262605968","created":"2024-06-01T00:45:32.072846+0000","modified":"2024-06-01T00:51:04.835598+0000","last_up_change":"2024-06-01T00:51:03.641391+0000","last_in_change":"2024-06-01T00:50:44.074666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-01T00:49:51.569670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a46a931e-6c2b-457c-808f-90274056b4f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6803","nonce":2649674789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6805","nonce":2649674789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6809","nonce":2649674789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2649674789},{"type":"v1","addr":"172.21.15.79:6807","nonce":2649674789}]},"public_addr":"172.21.15.79:6803/2649674789","cluster_addr":"172.21.15.79:6805/2649674789","heartbeat_back_addr":"172.21.15.79:6809/2649674789","heartbeat_front_addr":"172.21.15.79:6807/2649674789","state":["exists","up"]},{"osd":1,"uuid":"ba57b192-9f40-405c-b51e-2ce8229d54d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6811","nonce":2565529362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6813","nonce":2565529362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6817","nonce":2565529362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":2565529362},{"type":"v1","addr":"172.21.15.79:6815","nonce":2565529362}]},"public_addr":"172.21.15.79:6811/2565529362","cluster_addr":"172.21.15.79:6813/2565529362","heartbeat_back_addr":"172.21.15.79:6817/2565529362","heartbeat_front_addr":"172.21.15.79:6815/2565529362","state":["exists","up"]},{"osd":2,"uuid":"9d75526b-189e-4b38-a3d2-8c26d77c70a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6819","nonce":2237394964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6821","nonce":2237394964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6825","nonce":2237394964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2237394964},{"type":"v1","addr":"172.21.15.79:6823","nonce":2237394964}]},"public_addr":"172.21.15.79:6819/2237394964","cluster_addr":"172.21.15.79:6821/2237394964","heartbeat_back_addr":"172.21.15.79:6825/2237394964","heartbeat_front_addr":"172.21.15.79:6823/2237394964","state":["exists","up"]},{"osd":3,"uuid":"6b28a6b8-616c-4949-9ef8-8197138f624f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6801","nonce":70125620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6803","nonce":70125620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6807","nonce":70125620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":70125620},{"type":"v1","addr":"172.21.15.160:6805","nonce":70125620}]},"public_addr":"172.21.15.160:6801/70125620","cluster_addr":"172.21.15.160:6803/70125620","heartbeat_back_addr":"172.21.15.160:6807/70125620","heartbeat_front_addr":"172.21.15.160:6805/70125620","state":["exists","up"]},{"osd":4,"uuid":"7adb728d-8dc0-4629-a8ef-03def9adf728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6809","nonce":3806998400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6811","nonce":3806998400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6815","nonce":3806998400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3806998400},{"type":"v1","addr":"172.21.15.160:6813","nonce":3806998400}]},"public_addr":"172.21.15.160:6809/3806998400","cluster_addr":"172.21.15.160:6811/3806998400","heartbeat_back_addr":"172.21.15.160:6815/3806998400","heartbeat_front_addr":"172.21.15.160:6813/3806998400","state":["exists","up"]},{"osd":5,"uuid":"da4423f0-17d2-4449-b054-c46711cc0786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6817","nonce":610560416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6819","nonce":610560416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6823","nonce":610560416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":610560416},{"type":"v1","addr":"172.21.15.160:6821","nonce":610560416}]},"public_addr":"172.21.15.160:6817/610560416","cluster_addr":"172.21.15.160:6819/610560416","heartbeat_back_addr":"172.21.15.160:6823/610560416","heartbeat_front_addr":"172.21.15.160:6821/610560416","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:48:50.800804+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:49:19.715204+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:49:47.430320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:50:10.851994+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:50:35.913170+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T00:51:01.266185+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1815731501":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3424688232":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3930502773":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/1853708063":"2024-06-02T00:46:27.100908+0000","172.21.15.79:0/3808174981":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6800/1904937508":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6801/616716500":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3503804565":"2024-06-02T00:46:27.100908+0000","172.21.15.79:0/51499797":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6800/616716500":"2024-06-02T00:45:56.058310+0000","172.21.15.79:0/3979551222":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6801/1904937508":"2024-06-02T00:46:27.100908+0000","172.21.15.79:6800/3983389754":"2024-06-02T00:47:35.256206+0000","172.21.15.79:0/1696807089":"2024-06-02T00:47:35.256206+0000","172.21.15.79:6801/3983389754":"2024-06-02T00:47:35.256206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7d1c06aef0 msgr2=0x7f7d1c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7d1c06aef0 0x7f7d1c06d3b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009230 tx=0x7f7d2c0058d0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ffd10 msgr2=0x7f7d301975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ffd10 0x7f7d301975f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000d6c0 tx=0x7f7d2000db80 comp rx=0 tx=0).stop 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 shutdown_connections 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7d1c06aef0 0x7f7d1c06d3b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7d300ffd10 0x7f7d301975f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 --2- 172.21.15.79:0/3853236069 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7d300ff360 0x7f7d301970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 >> 172.21.15.79:0/3853236069 conn(0x7f7d30075e10 msgr2=0x7f7d30102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 shutdown_connections 2024-06-01T00:51:24.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:24.994+0000 7f7d3834a640 1 -- 172.21.15.79:0/3853236069 wait complete. 2024-06-01T00:51:25.610 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-01T00:51:25.610 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-01T00:51:25.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-01T00:51:25.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-01T00:51:25.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-01T00:51:25.612 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-01T00:51:25.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:25 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3853236069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T00:51:25.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:25 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3853236069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T00:51:25.896 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:25.899 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:25.979 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:25.980 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:25.981 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:25.982 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:26.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:26 smithi079 ceph-mon[29356]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:26.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:26 smithi160 ceph-mon[36639]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:28.629 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:28 smithi079 ceph-mon[29356]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.764+0000 7f01bf9e7640 1 -- 172.21.15.79:0/2688009579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 msgr2=0x7f01b81028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.764+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/2688009579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81028e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009880 tx=0x7f01a802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.764+0000 7f01bf9e7640 1 -- 172.21.15.79:0/2688009579 shutdown_connections 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.764+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/2688009579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81028e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.764+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/2688009579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01b81084f0 0x7f01b81088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.764+0000 7f01bf9e7640 1 -- 172.21.15.79:0/2688009579 >> 172.21.15.79:0/2688009579 conn(0x7f01b80fdf50 msgr2=0x7f01b8100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 -- 172.21.15.79:0/2688009579 shutdown_connections 2024-06-01T00:51:28.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 -- 172.21.15.79:0/2688009579 wait complete. 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 Processor -- start 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 -- start start 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01b81084f0 0x7f01b8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b8198f30 con 0x7f01b8102460 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bf9e7640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b81990a0 con 0x7f01b81084f0 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bd783640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bd783640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49188/0 (socket says 172.21.15.79:49188) 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.765+0000 7f01bd783640 1 -- 172.21.15.79:0/1437849373 learned_addr learned my addr 172.21.15.79:0/1437849373 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:28.770 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01bcf82640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01b81084f0 0x7f01b8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01bd783640 1 -- 172.21.15.79:0/1437849373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01b81084f0 msgr2=0x7f01b8198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01bd783640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01b81084f0 0x7f01b8198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01bd783640 1 -- 172.21.15.79:0/1437849373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8009560 con 0x7f01b8102460 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01bd783640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81983f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f01b400c8a0 tx=0x7f01b400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01b4007b80 con 0x7f01b8102460 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01b4007ce0 con 0x7f01b8102460 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b819ce30 con 0x7f01b8102460 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01b4018810 con 0x7f01b8102460 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.766+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b819d380 con 0x7f01b8102460 2024-06-01T00:51:28.771 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.767+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f01b810f9c0 con 0x7f01b8102460 2024-06-01T00:51:28.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f01b4018970 con 0x7f01b8102460 2024-06-01T00:51:28.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01ae7fc640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f01a406aef0 0x7f01a406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:28.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f01b4089d20 con 0x7f01b8102460 2024-06-01T00:51:28.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01bcf82640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f01a406aef0 0x7f01a406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:28.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01ae7fc640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] conn(0x7f01a4070c80 0x7f01a40730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:28.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 --> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01a4073750 con 0x7f01a4070c80 2024-06-01T00:51:28.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b408f400 con 0x7f01b8102460 2024-06-01T00:51:28.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.768+0000 7f01bdf84640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] conn(0x7f01a4070c80 0x7f01a40730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:28.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.769+0000 7f01bcf82640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f01a406aef0 0x7f01a406d3b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f01a8000c00 tx=0x7f01a8004120 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:28.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.769+0000 7f01bdf84640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] conn(0x7f01a4070c80 0x7f01a40730a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:28.774 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.769+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== osd.2 v2:172.21.15.79:6818/2237394964 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f01a4073750 con 0x7f01a4070c80 2024-06-01T00:51:28.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 --> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f01b8103ba0 con 0x7f01a4070c80 2024-06-01T00:51:28.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01ae7fc640 1 -- 172.21.15.79:0/1437849373 <== osd.2 v2:172.21.15.79:6818/2237394964 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f01b8103ba0 con 0x7f01a4070c80 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] conn(0x7f01a4070c80 msgr2=0x7f01a40730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] conn(0x7f01a4070c80 0x7f01a40730a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f01a406aef0 msgr2=0x7f01a406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f01a406aef0 0x7f01a406d3b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f01a8000c00 tx=0x7f01a8004120 comp rx=0 tx=0).stop 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 msgr2=0x7f01b81983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81983f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f01b400c8a0 tx=0x7f01b400cd60 comp rx=0 tx=0).stop 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 shutdown_connections 2024-06-01T00:51:28.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6818/2237394964,v1:172.21.15.79:6819/2237394964] conn(0x7f01a4070c80 0x7f01a40730a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f01a406aef0 0x7f01a406d3b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f01b81084f0 0x7f01b8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 --2- 172.21.15.79:0/1437849373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f01b8102460 0x7f01b81983f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:28.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.784+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 >> 172.21.15.79:0/1437849373 conn(0x7f01b80fdf50 msgr2=0x7f01b80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:28.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.785+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 shutdown_connections 2024-06-01T00:51:28.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:28.785+0000 7f01bf9e7640 1 -- 172.21.15.79:0/1437849373 wait complete. 2024-06-01T00:51:28.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:28 smithi160 ceph-mon[36639]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:29.951 INFO:teuthology.orchestra.run.smithi079.stdout:81604378647 2024-06-01T00:51:29.952 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-01T00:51:30.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 -- 172.21.15.79:0/603338098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ff380 msgr2=0x7f82bc0ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:30.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 --2- 172.21.15.79:0/603338098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ff380 0x7f82bc0ff760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009880 tx=0x7f82ac02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 -- 172.21.15.79:0/603338098 shutdown_connections 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 --2- 172.21.15.79:0/603338098 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f82bc0ffd30 0x7f82bc103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 --2- 172.21.15.79:0/603338098 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ff380 0x7f82bc0ff760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 -- 172.21.15.79:0/603338098 >> 172.21.15.79:0/603338098 conn(0x7f82bc0fac30 msgr2=0x7f82bc0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 -- 172.21.15.79:0/603338098 shutdown_connections 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.135+0000 7f82c27c5640 1 -- 172.21.15.79:0/603338098 wait complete. 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82c27c5640 1 Processor -- start 2024-06-01T00:51:30.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82c27c5640 1 -- start start 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82c27c5640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f82bc0ff380 0x7f82bc1962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82c27c5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ffd30 0x7f82bc196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82c27c5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc196eb0 con 0x7f82bc0ffd30 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82c27c5640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc19abd0 con 0x7f82bc0ff380 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ffd30 0x7f82bc196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ffd30 0x7f82bc196820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49194/0 (socket says 172.21.15.79:49194) 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82bb7fe640 1 -- 172.21.15.79:0/330287125 learned_addr learned my addr 172.21.15.79:0/330287125 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82bbfff640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f82bc0ff380 0x7f82bc1962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82bb7fe640 1 -- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f82bc0ff380 msgr2=0x7f82bc1962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82bb7fe640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f82bc0ff380 0x7f82bc1962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.136+0000 7f82bb7fe640 1 -- 172.21.15.79:0/330287125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac009560 con 0x7f82bc0ffd30 2024-06-01T00:51:30.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.137+0000 7f82bb7fe640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ffd30 0x7f82bc196820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f82b000b3d0 tx=0x7f82b000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:30.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.137+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b0011800 con 0x7f82bc0ffd30 2024-06-01T00:51:30.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.137+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82b0011e20 con 0x7f82bc0ffd30 2024-06-01T00:51:30.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.137+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc19aeb0 con 0x7f82bc0ffd30 2024-06-01T00:51:30.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.137+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b000f2e0 con 0x7f82bc0ffd30 2024-06-01T00:51:30.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.137+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc19b400 con 0x7f82bc0ffd30 2024-06-01T00:51:30.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.138+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82bc10c730 con 0x7f82bc0ffd30 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.138+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f82b0010380 con 0x7f82bc0ffd30 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82b97fa640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f82a406aef0 0x7f82a406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f82b00897e0 con 0x7f82bc0ffd30 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82bbfff640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f82a406aef0 0x7f82a406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82b97fa640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] conn(0x7f82a4070c80 0x7f82a40730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 --> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82a4073750 con 0x7f82a4070c80 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82b008e2a0 con 0x7f82bc0ffd30 2024-06-01T00:51:30.142 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82bbfff640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f82a406aef0 0x7f82a406d3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f82ac000c00 tx=0x7f82ac005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:30.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.139+0000 7f82c0d62640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] conn(0x7f82a4070c80 0x7f82a40730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:30.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.140+0000 7f82c0d62640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] conn(0x7f82a4070c80 0x7f82a40730a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:30.143 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.140+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== osd.3 v2:172.21.15.160:6800/70125620 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f82a4073750 con 0x7f82a4070c80 2024-06-01T00:51:30.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.154+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 --> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f82bc19b9d0 con 0x7f82a4070c80 2024-06-01T00:51:30.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.154+0000 7f82b97fa640 1 -- 172.21.15.79:0/330287125 <== osd.3 v2:172.21.15.160:6800/70125620 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f82bc19b9d0 con 0x7f82a4070c80 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] conn(0x7f82a4070c80 msgr2=0x7f82a40730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] conn(0x7f82a4070c80 0x7f82a40730a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f82a406aef0 msgr2=0x7f82a406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f82a406aef0 0x7f82a406d3b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f82ac000c00 tx=0x7f82ac005a20 comp rx=0 tx=0).stop 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ffd30 msgr2=0x7f82bc196820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ffd30 0x7f82bc196820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f82b000b3d0 tx=0x7f82b000b890 comp rx=0 tx=0).stop 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 shutdown_connections 2024-06-01T00:51:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:6800/70125620,v1:172.21.15.160:6801/70125620] conn(0x7f82a4070c80 0x7f82a40730a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f82a406aef0 0x7f82a406d3b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f82bc0ffd30 0x7f82bc196820 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 --2- 172.21.15.79:0/330287125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f82bc0ff380 0x7f82bc1962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 >> 172.21.15.79:0/330287125 conn(0x7f82bc0fac30 msgr2=0x7f82bc109520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 shutdown_connections 2024-06-01T00:51:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:30.155+0000 7f82c27c5640 1 -- 172.21.15.79:0/330287125 wait complete. 2024-06-01T00:51:30.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:30 smithi079 ceph-mon[29356]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:30.683 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:30 smithi160 ceph-mon[36639]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:32.514 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:32 smithi079 ceph-mon[29356]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:32 smithi160 ceph-mon[36639]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:33.689 INFO:teuthology.orchestra.run.smithi079.stdout:115964117010 2024-06-01T00:51:33.689 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-01T00:51:34.677 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:34 smithi079 ceph-mon[29356]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:34.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:34 smithi160 ceph-mon[36639]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:34.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 -- 172.21.15.79:0/928073006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ff3a0 msgr2=0x7ff7740ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:34.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 --2- 172.21.15.79:0/928073006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ff3a0 0x7ff7740ff780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff770009880 tx=0x7ff77002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 -- 172.21.15.79:0/928073006 shutdown_connections 2024-06-01T00:51:34.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 --2- 172.21.15.79:0/928073006 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7740ffd50 0x7ff774103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 --2- 172.21.15.79:0/928073006 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ff3a0 0x7ff7740ff780 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 -- 172.21.15.79:0/928073006 >> 172.21.15.79:0/928073006 conn(0x7ff7740fac50 msgr2=0x7ff7740fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 -- 172.21.15.79:0/928073006 shutdown_connections 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 -- 172.21.15.79:0/928073006 wait complete. 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 Processor -- start 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.796+0000 7ff77bd81640 1 -- start start 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77bd81640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7740ff3a0 0x7ff7741962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77bd81640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ffd50 0x7ff7741967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77bd81640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff774196de0 con 0x7ff7740ffd50 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77bd81640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff774196f50 con 0x7ff7740ff3a0 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77931c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ffd50 0x7ff7741967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77931c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ffd50 0x7ff7741967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47562/0 (socket says 172.21.15.79:47562) 2024-06-01T00:51:34.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77931c640 1 -- 172.21.15.79:0/2449566950 learned_addr learned my addr 172.21.15.79:0/2449566950 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77931c640 1 -- 172.21.15.79:0/2449566950 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7740ff3a0 msgr2=0x7ff7741962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77931c640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7740ff3a0 0x7ff7741962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77931c640 1 -- 172.21.15.79:0/2449566950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff770009560 con 0x7ff7740ffd50 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77931c640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ffd50 0x7ff7741967e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff76400c8f0 tx=0x7ff76400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff764004410 con 0x7ff7740ffd50 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff764007c60 con 0x7ff7740ffd50 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77419ace0 con 0x7ff7740ffd50 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff76400f450 con 0x7ff7740ffd50 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.797+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77419b230 con 0x7ff7740ffd50 2024-06-01T00:51:34.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.798+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff77410c650 con 0x7ff7740ffd50 2024-06-01T00:51:34.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ff7640077a0 con 0x7ff7740ffd50 2024-06-01T00:51:34.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff76affd640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff76006aef0 0x7ff76006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:34.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff764089ac0 con 0x7ff7740ffd50 2024-06-01T00:51:34.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff779b1d640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff76006aef0 0x7ff76006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:34.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff76affd640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] conn(0x7ff760070c80 0x7ff7600730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:34.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 --> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff760073750 con 0x7ff760070c80 2024-06-01T00:51:34.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff76408e2a0 con 0x7ff7740ffd50 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.799+0000 7ff77a31e640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] conn(0x7ff760070c80 0x7ff7600730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.800+0000 7ff779b1d640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff76006aef0 0x7ff76006d3b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff770009880 tx=0x7ff770000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.800+0000 7ff77a31e640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] conn(0x7ff760070c80 0x7ff7600730a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.800+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== osd.0 v2:172.21.15.79:6802/2649674789 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ff760073750 con 0x7ff760070c80 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 --> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff77419b740 con 0x7ff760070c80 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff76affd640 1 -- 172.21.15.79:0/2449566950 <== osd.0 v2:172.21.15.79:6802/2649674789 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff77419b740 con 0x7ff760070c80 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] conn(0x7ff760070c80 msgr2=0x7ff7600730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff77bd81640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] conn(0x7ff760070c80 0x7ff7600730a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff76006aef0 msgr2=0x7ff76006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff77bd81640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff76006aef0 0x7ff76006d3b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff770009880 tx=0x7ff770000bc0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ffd50 msgr2=0x7ff7741967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.814+0000 7ff77bd81640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ffd50 0x7ff7741967e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff76400c8f0 tx=0x7ff76400cdb0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 shutdown_connections 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6802/2649674789,v1:172.21.15.79:6803/2649674789] conn(0x7ff760070c80 0x7ff7600730a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff76006aef0 0x7ff76006d3b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff7740ffd50 0x7ff7741967e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 --2- 172.21.15.79:0/2449566950 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff7740ff3a0 0x7ff7741962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 >> 172.21.15.79:0/2449566950 conn(0x7ff7740fac50 msgr2=0x7ff774068cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 shutdown_connections 2024-06-01T00:51:34.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.815+0000 7ff77bd81640 1 -- 172.21.15.79:0/2449566950 wait complete. 2024-06-01T00:51:35.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:35.743 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:35.990 INFO:teuthology.orchestra.run.smithi079.stdout:38654705700 2024-06-01T00:51:35.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 -- 172.21.15.79:0/3384058077 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6900100e60 msgr2=0x7f6900104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 --2- 172.21.15.79:0/3384058077 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6900100e60 0x7f6900104eb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009880 tx=0x7f68f402a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 -- 172.21.15.79:0/3384058077 shutdown_connections 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 --2- 172.21.15.79:0/3384058077 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6900100e60 0x7f6900104eb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 --2- 172.21.15.79:0/3384058077 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f69001004b0 0x7f6900100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 -- 172.21.15.79:0/3384058077 >> 172.21.15.79:0/3384058077 conn(0x7f69000fbd40 msgr2=0x7f69000fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 -- 172.21.15.79:0/3384058077 shutdown_connections 2024-06-01T00:51:36.137 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 -- 172.21.15.79:0/3384058077 wait complete. 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 Processor -- start 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.671+0000 7f69064ad640 1 -- start start 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f69064ad640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69001004b0 0x7f69000727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f69064ad640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6900100e60 0x7f6900072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f69064ad640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6900073230 con 0x7f69001004b0 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f69064ad640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69000733a0 con 0x7f6900100e60 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69001004b0 0x7f69000727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69001004b0 0x7f69000727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47584/0 (socket says 172.21.15.79:47584) 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fffff640 1 -- 172.21.15.79:0/711834138 learned_addr learned my addr 172.21.15.79:0/711834138 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fffff640 1 -- 172.21.15.79:0/711834138 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6900100e60 msgr2=0x7f6900072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fffff640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6900100e60 0x7f6900072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fffff640 1 -- 172.21.15.79:0/711834138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f4009560 con 0x7f69001004b0 2024-06-01T00:51:36.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fffff640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69001004b0 0x7f69000727b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f68f000b420 tx=0x7f68f000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68f0011800 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69001a2540 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68f0011e20 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68f000f2e0 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.672+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69001a28a0 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.673+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f690010d840 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.674+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f68f0011960 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.674+0000 7f68fd7fa640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f68e806aef0 0x7f68e806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.674+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f68f0089560 con 0x7f69001004b0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.674+0000 7f68ff7fe640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f68e806aef0 0x7f68e806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.674+0000 7f68fd7fa640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] conn(0x7f68e8070c80 0x7f68e80730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.674+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 --> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68e8073750 con 0x7f68e8070c80 2024-06-01T00:51:36.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.674+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68f008bc30 con 0x7f69001004b0 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.675+0000 7f6904a4a640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] conn(0x7f68e8070c80 0x7f68e80730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.675+0000 7f68ff7fe640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f68e806aef0 0x7f68e806d3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6900073ee0 tx=0x7f68f4000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.675+0000 7f6904a4a640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] conn(0x7f68e8070c80 0x7f68e80730a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.676+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== osd.1 v2:172.21.15.79:6810/2565529362 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f68e8073750 con 0x7f68e8070c80 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.689+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 --> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6900060b70 con 0x7f68e8070c80 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.689+0000 7f68fd7fa640 1 -- 172.21.15.79:0/711834138 <== osd.1 v2:172.21.15.79:6810/2565529362 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6900060b70 con 0x7f68e8070c80 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] conn(0x7f68e8070c80 msgr2=0x7f68e80730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.140 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] conn(0x7f68e8070c80 0x7f68e80730a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f68e806aef0 msgr2=0x7f68e806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f68e806aef0 0x7f68e806d3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6900073ee0 tx=0x7f68f4000bc0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69001004b0 msgr2=0x7f69000727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69001004b0 0x7f69000727b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f68f000b420 tx=0x7f68f000b8e0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 shutdown_connections 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f68e806aef0 0x7f68e806d3b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:6810/2565529362,v1:172.21.15.79:6811/2565529362] conn(0x7f68e8070c80 0x7f68e80730a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6900100e60 0x7f6900072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 --2- 172.21.15.79:0/711834138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69001004b0 0x7f69000727b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 >> 172.21.15.79:0/711834138 conn(0x7f69000fbd40 msgr2=0x7f69000fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 shutdown_connections 2024-06-01T00:51:36.141 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:35.690+0000 7f69064ad640 1 -- 172.21.15.79:0/711834138 wait complete. 2024-06-01T00:51:36.512 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 -- 172.21.15.79:0/608804951 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100470 msgr2=0x7f1868100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 --2- 172.21.15.79:0/608804951 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100470 0x7f1868100850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1858009880 tx=0x7f185802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 -- 172.21.15.79:0/608804951 shutdown_connections 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 --2- 172.21.15.79:0/608804951 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1868100e20 0x7f1868104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 --2- 172.21.15.79:0/608804951 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100470 0x7f1868100850 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 -- 172.21.15.79:0/608804951 >> 172.21.15.79:0/608804951 conn(0x7f18680fbd20 msgr2=0x7f18680fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 -- 172.21.15.79:0/608804951 shutdown_connections 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.675+0000 7f186e860640 1 -- 172.21.15.79:0/608804951 wait complete. 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f186e860640 1 Processor -- start 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f186e860640 1 -- start start 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f186e860640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1868100470 0x7f1868198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.513 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f186e860640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100e20 0x7f1868198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f186e860640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1868199110 con 0x7f1868100e20 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f186e860640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186819ce30 con 0x7f1868100470 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f18677fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100e20 0x7f1868198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f18677fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100e20 0x7f1868198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47542/0 (socket says 172.21.15.79:47542) 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f18677fe640 1 -- 172.21.15.79:0/295816717 learned_addr learned my addr 172.21.15.79:0/295816717 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.676+0000 7f1867fff640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1868100470 0x7f1868198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f18677fe640 1 -- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1868100470 msgr2=0x7f1868198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f18677fe640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1868100470 0x7f1868198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f18677fe640 1 -- 172.21.15.79:0/295816717 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858009560 con 0x7f1868100e20 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f1867fff640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1868100470 0x7f1868198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f18677fe640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100e20 0x7f1868198a80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f185c00b3d0 tx=0x7f185c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:36.514 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185c011800 con 0x7f1868100e20 2024-06-01T00:51:36.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f185c011e20 con 0x7f1868100e20 2024-06-01T00:51:36.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185c00f2e0 con 0x7f1868100e20 2024-06-01T00:51:36.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186819d110 con 0x7f1868100e20 2024-06-01T00:51:36.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.677+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186819d660 con 0x7f1868100e20 2024-06-01T00:51:36.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.678+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f186810d820 con 0x7f1868100e20 2024-06-01T00:51:36.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.678+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f185c010380 con 0x7f1868100e20 2024-06-01T00:51:36.515 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f18657fa640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f185006aef0 0x7f185006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f185c0897e0 con 0x7f1868100e20 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f1867fff640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f185006aef0 0x7f185006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f18657fa640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] conn(0x7f1850070c80 0x7f18500730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 --> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1850073750 con 0x7f1850070c80 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f185c08e2a0 con 0x7f1868100e20 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f1867fff640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f185006aef0 0x7f185006d3b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1858000c00 tx=0x7f1858005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.679+0000 7f186cdfd640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] conn(0x7f1850070c80 0x7f18500730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.680+0000 7f186cdfd640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] conn(0x7f1850070c80 0x7f18500730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.681+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== osd.5 v2:172.21.15.160:6816/610560416 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f1850073750 con 0x7f1850070c80 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.694+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 --> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f186819dbf0 con 0x7f1850070c80 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f18657fa640 1 -- 172.21.15.79:0/295816717 <== osd.5 v2:172.21.15.160:6816/610560416 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f186819dbf0 con 0x7f1850070c80 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] conn(0x7f1850070c80 msgr2=0x7f18500730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.516 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] conn(0x7f1850070c80 0x7f18500730a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f185006aef0 msgr2=0x7f185006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f185006aef0 0x7f185006d3b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1858000c00 tx=0x7f1858005a20 comp rx=0 tx=0).stop 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100e20 msgr2=0x7f1868198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100e20 0x7f1868198a80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f185c00b3d0 tx=0x7f185c00b890 comp rx=0 tx=0).stop 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 shutdown_connections 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:6816/610560416,v1:172.21.15.160:6817/610560416] conn(0x7f1850070c80 0x7f18500730a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f185006aef0 0x7f185006d3b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1868100e20 0x7f1868198a80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 --2- 172.21.15.79:0/295816717 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1868100470 0x7f1868198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:36.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 >> 172.21.15.79:0/295816717 conn(0x7f18680fbd20 msgr2=0x7f18680fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:36.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 shutdown_connections 2024-06-01T00:51:36.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:34.695+0000 7f186e860640 1 -- 172.21.15.79:0/295816717 wait complete. 2024-06-01T00:51:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:36 smithi160 ceph-mon[36639]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:36.830 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:36 smithi079 ceph-mon[29356]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:37.866 INFO:teuthology.orchestra.run.smithi079.stdout:171798691850 2024-06-01T00:51:37.866 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-01T00:51:37.881 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:38.292 INFO:teuthology.orchestra.run.smithi079.stdout:60129542174 2024-06-01T00:51:38.293 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-01T00:51:38.599 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:38 smithi079 ceph-mon[29356]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:38 smithi160 ceph-mon[36639]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:39.831 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:39.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 -- 172.21.15.79:0/1652834446 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100e30 msgr2=0x7fab24104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:39.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 --2- 172.21.15.79:0/1652834446 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100e30 0x7fab24104e80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fab18009880 tx=0x7fab1802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 -- 172.21.15.79:0/1652834446 shutdown_connections 2024-06-01T00:51:39.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 --2- 172.21.15.79:0/1652834446 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100e30 0x7fab24104e80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 --2- 172.21.15.79:0/1652834446 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab24100480 0x7fab24100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 -- 172.21.15.79:0/1652834446 >> 172.21.15.79:0/1652834446 conn(0x7fab240fbd50 msgr2=0x7fab240fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:39.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 -- 172.21.15.79:0/1652834446 shutdown_connections 2024-06-01T00:51:39.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 -- 172.21.15.79:0/1652834446 wait complete. 2024-06-01T00:51:39.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.841+0000 7fab29d25640 1 Processor -- start 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab29d25640 1 -- start start 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab29d25640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100480 0x7fab241962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab29d25640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab24100e30 0x7fab241967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab29d25640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab24196de0 con 0x7fab24100480 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab29d25640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab24196f50 con 0x7fab24100e30 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab237fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100480 0x7fab241962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab237fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100480 0x7fab241962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47606/0 (socket says 172.21.15.79:47606) 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab237fe640 1 -- 172.21.15.79:0/1130445579 learned_addr learned my addr 172.21.15.79:0/1130445579 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.842+0000 7fab22ffd640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab24100e30 0x7fab241967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab237fe640 1 -- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab24100e30 msgr2=0x7fab241967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:39.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab237fe640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab24100e30 0x7fab241967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab237fe640 1 -- 172.21.15.79:0/1130445579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab18009560 con 0x7fab24100480 2024-06-01T00:51:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab237fe640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100480 0x7fab241962a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c8a0 tx=0x7fab1400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab14007b80 con 0x7fab24100480 2024-06-01T00:51:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2419ace0 con 0x7fab24100480 2024-06-01T00:51:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab14007ce0 con 0x7fab24100480 2024-06-01T00:51:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab14018810 con 0x7fab24100480 2024-06-01T00:51:39.846 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.843+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2419b200 con 0x7fab24100480 2024-06-01T00:51:39.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.844+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fab2410d710 con 0x7fab24100480 2024-06-01T00:51:39.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.845+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fab1400f450 con 0x7fab24100480 2024-06-01T00:51:39.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.845+0000 7fab20ff9640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab0c06aef0 0x7fab0c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:39.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.845+0000 7fab22ffd640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab0c06aef0 0x7fab0c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:39.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.845+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fab14089cc0 con 0x7fab24100480 2024-06-01T00:51:39.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.845+0000 7fab20ff9640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] conn(0x7fab0c070c80 0x7fab0c0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:39.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.845+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 --> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab0c073750 con 0x7fab0c070c80 2024-06-01T00:51:39.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.845+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab1408d020 con 0x7fab24100480 2024-06-01T00:51:39.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.846+0000 7fab23fff640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] conn(0x7fab0c070c80 0x7fab0c0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:39.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.846+0000 7fab22ffd640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab0c06aef0 0x7fab0c06d3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fab18005940 tx=0x7fab180058d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:39.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.847+0000 7fab23fff640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] conn(0x7fab0c070c80 0x7fab0c0730a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:39.848 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.847+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== osd.4 v2:172.21.15.160:6808/3806998400 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fab0c073750 con 0x7fab0c070c80 2024-06-01T00:51:39.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.864+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 --> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fab24101960 con 0x7fab0c070c80 2024-06-01T00:51:39.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.864+0000 7fab20ff9640 1 -- 172.21.15.79:0/1130445579 <== osd.4 v2:172.21.15.160:6808/3806998400 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fab24101960 con 0x7fab0c070c80 2024-06-01T00:51:39.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] conn(0x7fab0c070c80 msgr2=0x7fab0c0730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:39.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] conn(0x7fab0c070c80 0x7fab0c0730a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab0c06aef0 msgr2=0x7fab0c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab0c06aef0 0x7fab0c06d3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fab18005940 tx=0x7fab180058d0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100480 msgr2=0x7fab241962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100480 0x7fab241962a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c8a0 tx=0x7fab1400cd60 comp rx=0 tx=0).stop 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 shutdown_connections 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:6808/3806998400,v1:172.21.15.160:6809/3806998400] conn(0x7fab0c070c80 0x7fab0c0730a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab0c06aef0 0x7fab0c06d3b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab24100e30 0x7fab241967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 --2- 172.21.15.79:0/1130445579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab24100480 0x7fab241962a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.865+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 >> 172.21.15.79:0/1130445579 conn(0x7fab240fbd50 msgr2=0x7fab240fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.866+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 shutdown_connections 2024-06-01T00:51:39.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:39.866+0000 7fab29d25640 1 -- 172.21.15.79:0/1130445579 wait complete. 2024-06-01T00:51:40.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:40 smithi079 ceph-mon[29356]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:40.803 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:40 smithi160 ceph-mon[36639]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:41.743 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:41.743 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:42.183 INFO:teuthology.orchestra.run.smithi079.stdout:150323855375 2024-06-01T00:51:42.184 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-01T00:51:42.465 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:42 smithi079 ceph-mon[29356]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- 172.21.15.79:0/604694776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 msgr2=0x7faf380ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/604694776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf380ff740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faf28009880 tx=0x7faf2802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- 172.21.15.79:0/604694776 shutdown_connections 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/604694776 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf380ffd10 0x7faf38103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/604694776 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf380ff740 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- 172.21.15.79:0/604694776 >> 172.21.15.79:0/604694776 conn(0x7faf38075e10 msgr2=0x7faf380fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- 172.21.15.79:0/604694776 shutdown_connections 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- 172.21.15.79:0/604694776 wait complete. 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 Processor -- start 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- start start 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf38196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf380ffd10 0x7faf38196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf38196ef0 con 0x7faf380ff360 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.724+0000 7faf3ec2c640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3819ac10 con 0x7faf380ffd10 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3c9c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf38196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3c9c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf38196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43052/0 (socket says 172.21.15.79:43052) 2024-06-01T00:51:42.727 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3c9c8640 1 -- 172.21.15.79:0/1155180420 learned_addr learned my addr 172.21.15.79:0/1155180420 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf37fff640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf380ffd10 0x7faf38196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3c9c8640 1 -- 172.21.15.79:0/1155180420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf380ffd10 msgr2=0x7faf38196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3c9c8640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf380ffd10 0x7faf38196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3c9c8640 1 -- 172.21.15.79:0/1155180420 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28009560 con 0x7faf380ff360 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3c9c8640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf38196320 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faf2802a9c0 tx=0x7faf280055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf35ffb640 1 -- 172.21.15.79:0/1155180420 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf28004aa0 con 0x7faf380ff360 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf35ffb640 1 -- 172.21.15.79:0/1155180420 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf28004c00 con 0x7faf380ff360 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3819ae90 con 0x7faf380ff360 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.725+0000 7faf35ffb640 1 -- 172.21.15.79:0/1155180420 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf28003ab0 con 0x7faf380ff360 2024-06-01T00:51:42.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.726+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3819b380 con 0x7faf380ff360 2024-06-01T00:51:42.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.726+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf38100840 con 0x7faf380ff360 2024-06-01T00:51:42.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.727+0000 7faf35ffb640 1 -- 172.21.15.79:0/1155180420 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7faf28031d20 con 0x7faf380ff360 2024-06-01T00:51:42.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.727+0000 7faf35ffb640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7faf2006aef0 0x7faf2006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:42.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.727+0000 7faf35ffb640 1 -- 172.21.15.79:0/1155180420 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7faf280aac10 con 0x7faf380ff360 2024-06-01T00:51:42.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.727+0000 7faf37fff640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7faf2006aef0 0x7faf2006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:42.729 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.728+0000 7faf37fff640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7faf2006aef0 0x7faf2006d3b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf38197880 tx=0x7faf2c0094f0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:42.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.732+0000 7faf35ffb640 1 -- 172.21.15.79:0/1155180420 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7faf28076270 con 0x7faf380ff360 2024-06-01T00:51:42.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:42 smithi160 ceph-mon[36639]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:42.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.908+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faf3804e210 con 0x7faf380ff360 2024-06-01T00:51:42.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.908+0000 7faf35ffb640 1 -- 172.21.15.79:0/1155180420 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faf28079890 con 0x7faf380ff360 2024-06-01T00:51:42.910 INFO:teuthology.orchestra.run.smithi079.stdout:81604378650 2024-06-01T00:51:42.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.910+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7faf2006aef0 msgr2=0x7faf2006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.910+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7faf2006aef0 0x7faf2006d3b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf38197880 tx=0x7faf2c0094f0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.910+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 msgr2=0x7faf38196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.910+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf38196320 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faf2802a9c0 tx=0x7faf280055a0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.911+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 shutdown_connections 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.911+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7faf2006aef0 0x7faf2006d3b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.911+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7faf380ffd10 0x7faf38196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.911+0000 7faf3ec2c640 1 --2- 172.21.15.79:0/1155180420 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faf380ff360 0x7faf38196320 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.911+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 >> 172.21.15.79:0/1155180420 conn(0x7faf38075e10 msgr2=0x7faf380fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.911+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 shutdown_connections 2024-06-01T00:51:42.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.911+0000 7faf3ec2c640 1 -- 172.21.15.79:0/1155180420 wait complete. 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 -- 172.21.15.79:0/2618120708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 msgr2=0x7fce4410a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/2618120708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce4410a230 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fce38009880 tx=0x7fce3802a530 comp rx=0 tx=0).stop 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 -- 172.21.15.79:0/2618120708 shutdown_connections 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/2618120708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce4410a230 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/2618120708 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce44101420 0x7fce44101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 -- 172.21.15.79:0/2618120708 >> 172.21.15.79:0/2618120708 conn(0x7fce44075e10 msgr2=0x7fce440fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 -- 172.21.15.79:0/2618120708 shutdown_connections 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.967+0000 7fce4c7f3640 1 -- 172.21.15.79:0/2618120708 wait complete. 2024-06-01T00:51:42.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce4c7f3640 1 Processor -- start 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce4c7f3640 1 -- start start 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce4c7f3640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce44101420 0x7fce44193fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce4c7f3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce441944f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce4c7f3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4419b090 con 0x7fce44101d40 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce4c7f3640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4419b200 con 0x7fce44101420 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce49d8e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce441944f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce49d8e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce441944f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43068/0 (socket says 172.21.15.79:43068) 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.968+0000 7fce49d8e640 1 -- 172.21.15.79:0/3393446309 learned_addr learned my addr 172.21.15.79:0/3393446309 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce4a58f640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce44101420 0x7fce44193fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce49d8e640 1 -- 172.21.15.79:0/3393446309 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce44101420 msgr2=0x7fce44193fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce49d8e640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce44101420 0x7fce44193fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce49d8e640 1 -- 172.21.15.79:0/3393446309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38009560 con 0x7fce44101d40 2024-06-01T00:51:42.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce49d8e640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce441944f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fce38006010 tx=0x7fce380055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce437fe640 1 -- 172.21.15.79:0/3393446309 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce38004aa0 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce44194b40 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce437fe640 1 -- 172.21.15.79:0/3393446309 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce38004c00 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce437fe640 1 -- 172.21.15.79:0/3393446309 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce38003a20 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.969+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce44198c40 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.970+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce440684b0 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.971+0000 7fce437fe640 1 -- 172.21.15.79:0/3393446309 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fce3802ad50 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.971+0000 7fce437fe640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fce2c06af00 0x7fce2c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.971+0000 7fce437fe640 1 -- 172.21.15.79:0/3393446309 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fce380aaad0 con 0x7fce44101d40 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.971+0000 7fce4a58f640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fce2c06af00 0x7fce2c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.972+0000 7fce4a58f640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fce2c06af00 0x7fce2c06d3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce3400a400 tx=0x7fce3400d040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:42.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:42.975+0000 7fce437fe640 1 -- 172.21.15.79:0/3393446309 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fce38076160 con 0x7fce44101d40 2024-06-01T00:51:43.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.156+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fce44195490 con 0x7fce44101d40 2024-06-01T00:51:43.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.156+0000 7fce437fe640 1 -- 172.21.15.79:0/3393446309 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fce38079780 con 0x7fce44101d40 2024-06-01T00:51:43.157 INFO:teuthology.orchestra.run.smithi079.stdout:38654705701 2024-06-01T00:51:43.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fce2c06af00 msgr2=0x7fce2c06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:43.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fce2c06af00 0x7fce2c06d3c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce3400a400 tx=0x7fce3400d040 comp rx=0 tx=0).stop 2024-06-01T00:51:43.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 msgr2=0x7fce441944f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:43.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce441944f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fce38006010 tx=0x7fce380055a0 comp rx=0 tx=0).stop 2024-06-01T00:51:43.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 shutdown_connections 2024-06-01T00:51:43.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fce2c06af00 0x7fce2c06d3c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:43.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce44101d40 0x7fce441944f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:43.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 --2- 172.21.15.79:0/3393446309 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fce44101420 0x7fce44193fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:43.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 >> 172.21.15.79:0/3393446309 conn(0x7fce44075e10 msgr2=0x7fce440fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:43.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 shutdown_connections 2024-06-01T00:51:43.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:43.158+0000 7fce4c7f3640 1 -- 172.21.15.79:0/3393446309 wait complete. 2024-06-01T00:51:43.514 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:43 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1155180420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-01T00:51:43.514 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:43 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3393446309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-01T00:51:43.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:43 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1155180420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-01T00:51:43.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:43 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3393446309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-01T00:51:44.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705701 for osd.0 2024-06-01T00:51:44.265 DEBUG:teuthology.parallel:result is None 2024-06-01T00:51:44.444 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:44.555 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:44 smithi079 ceph-mon[29356]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:44 smithi160 ceph-mon[36639]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:44.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378647 got 81604378650 for osd.2 2024-06-01T00:51:44.943 DEBUG:teuthology.parallel:result is None 2024-06-01T00:51:46.000 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:46.722 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:46 smithi079 ceph-mon[29356]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:46.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 -- 172.21.15.79:0/2586841934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 msgr2=0x7fa5bc0ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/2586841934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc0ff740 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4009c20 tx=0x7fa5b402a970 comp rx=0 tx=0).stop 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 -- 172.21.15.79:0/2586841934 shutdown_connections 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/2586841934 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5bc0ffd10 0x7fa5bc103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/2586841934 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc0ff740 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 -- 172.21.15.79:0/2586841934 >> 172.21.15.79:0/2586841934 conn(0x7fa5bc075e10 msgr2=0x7fa5bc0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 -- 172.21.15.79:0/2586841934 shutdown_connections 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 -- 172.21.15.79:0/2586841934 wait complete. 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.721+0000 7fa5c1f70640 1 Processor -- start 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5c1f70640 1 -- start start 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5c1f70640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5c1f70640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5bc0ffd10 0x7fa5bc196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:46.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5c1f70640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc196d40 con 0x7fa5bc0ff360 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5c1f70640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc196eb0 con 0x7fa5bc0ffd10 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc196200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43080/0 (socket says 172.21.15.79:43080) 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5bb7fe640 1 -- 172.21.15.79:0/3690355364 learned_addr learned my addr 172.21.15.79:0/3690355364 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5baffd640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5bc0ffd10 0x7fa5bc196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5bb7fe640 1 -- 172.21.15.79:0/3690355364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5bc0ffd10 msgr2=0x7fa5bc196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5bb7fe640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5bc0ffd10 0x7fa5bc196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5bb7fe640 1 -- 172.21.15.79:0/3690355364 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b4009900 con 0x7fa5bc0ff360 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5bb7fe640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc196200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa5b402ae90 tx=0x7fa5b40308a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.722+0000 7fa5b8ff9640 1 -- 172.21.15.79:0/3690355364 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5b402f590 con 0x7fa5bc0ff360 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.723+0000 7fa5b8ff9640 1 -- 172.21.15.79:0/3690355364 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5b4030ca0 con 0x7fa5bc0ff360 2024-06-01T00:51:46.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.723+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc19abe0 con 0x7fa5bc0ff360 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.723+0000 7fa5b8ff9640 1 -- 172.21.15.79:0/3690355364 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5b4039680 con 0x7fa5bc0ff360 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.723+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc19b0d0 con 0x7fa5bc0ff360 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.724+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc100840 con 0x7fa5bc0ff360 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.724+0000 7fa5b8ff9640 1 -- 172.21.15.79:0/3690355364 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fa5b400a040 con 0x7fa5bc0ff360 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.724+0000 7fa5b8ff9640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa5a806aef0 0x7fa5a806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.725+0000 7fa5baffd640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa5a806aef0 0x7fa5a806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.725+0000 7fa5b8ff9640 1 -- 172.21.15.79:0/3690355364 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa5b402c040 con 0x7fa5bc0ff360 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.725+0000 7fa5baffd640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa5a806aef0 0x7fa5a806d3b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4005e10 tx=0x7fa5a4005da0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:46.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.729+0000 7fa5b8ff9640 1 -- 172.21.15.79:0/3690355364 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa5b40765f0 con 0x7fa5bc0ff360 2024-06-01T00:51:46.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:46 smithi160 ceph-mon[36639]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:46.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.903+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa5bc10c3c0 con 0x7fa5bc0ff360 2024-06-01T00:51:46.904 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.903+0000 7fa5b8ff9640 1 -- 172.21.15.79:0/3690355364 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa5b4079c10 con 0x7fa5bc0ff360 2024-06-01T00:51:46.904 INFO:teuthology.orchestra.run.smithi079.stdout:115964117013 2024-06-01T00:51:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa5a806aef0 msgr2=0x7fa5a806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa5a806aef0 0x7fa5a806d3b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4005e10 tx=0x7fa5a4005da0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 msgr2=0x7fa5bc196200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc196200 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa5b402ae90 tx=0x7fa5b40308a0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 shutdown_connections 2024-06-01T00:51:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa5a806aef0 0x7fa5a806d3b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.906 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa5bc0ffd10 0x7fa5bc196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 --2- 172.21.15.79:0/3690355364 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa5bc0ff360 0x7fa5bc196200 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 >> 172.21.15.79:0/3690355364 conn(0x7fa5bc075e10 msgr2=0x7fa5bc0fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 shutdown_connections 2024-06-01T00:51:46.907 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:46.905+0000 7fa5c1f70640 1 -- 172.21.15.79:0/3690355364 wait complete. 2024-06-01T00:51:47.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:47 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3690355364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-01T00:51:47.644 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117010 got 115964117013 for osd.3 2024-06-01T00:51:47.645 DEBUG:teuthology.parallel:result is None 2024-06-01T00:51:47.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:47 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3690355364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-01T00:51:48.763 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:48 smithi079 ceph-mon[29356]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:48.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:48 smithi160 ceph-mon[36639]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:49.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 -- 172.21.15.79:0/3859818783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 msgr2=0x7f642410a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:49.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 --2- 172.21.15.79:0/3859818783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f642410a290 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6418009880 tx=0x7f641802a530 comp rx=0 tx=0).stop 2024-06-01T00:51:49.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 -- 172.21.15.79:0/3859818783 shutdown_connections 2024-06-01T00:51:49.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 --2- 172.21.15.79:0/3859818783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f642410a290 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:49.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 --2- 172.21.15.79:0/3859818783 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6424101480 0x7f6424101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:49.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 -- 172.21.15.79:0/3859818783 >> 172.21.15.79:0/3859818783 conn(0x7f64240fac70 msgr2=0x7f64240fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:49.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 -- 172.21.15.79:0/3859818783 shutdown_connections 2024-06-01T00:51:49.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.071+0000 7f642beca640 1 -- 172.21.15.79:0/3859818783 wait complete. 2024-06-01T00:51:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f642beca640 1 Processor -- start 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f642beca640 1 -- start start 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f642beca640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6424101480 0x7f6424196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f642beca640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f64241968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f642beca640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6424196f50 con 0x7f6424101da0 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f642beca640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642419ac70 con 0x7f6424101480 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f6429465640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f64241968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f6429465640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f64241968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43104/0 (socket says 172.21.15.79:43104) 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.072+0000 7f6429465640 1 -- 172.21.15.79:0/1357092408 learned_addr learned my addr 172.21.15.79:0/1357092408 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6429465640 1 -- 172.21.15.79:0/1357092408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6424101480 msgr2=0x7f6424196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6429c66640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6424101480 0x7f6424196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6429465640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6424101480 0x7f6424196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6429465640 1 -- 172.21.15.79:0/1357092408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418009560 con 0x7f6424101da0 2024-06-01T00:51:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6429465640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f64241968c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f641802aa50 tx=0x7f64180055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6422ffd640 1 -- 172.21.15.79:0/1357092408 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6418004aa0 con 0x7f6424101da0 2024-06-01T00:51:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6422ffd640 1 -- 172.21.15.79:0/1357092408 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6418004c00 con 0x7f6424101da0 2024-06-01T00:51:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642419aef0 con 0x7f6424101da0 2024-06-01T00:51:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f6422ffd640 1 -- 172.21.15.79:0/1357092408 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6418003ab0 con 0x7f6424101da0 2024-06-01T00:51:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.073+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642419b3e0 con 0x7f6424101da0 2024-06-01T00:51:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.074+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6424068fe0 con 0x7f6424101da0 2024-06-01T00:51:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.074+0000 7f6422ffd640 1 -- 172.21.15.79:0/1357092408 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f641802ad50 con 0x7f6424101da0 2024-06-01T00:51:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.075+0000 7f6422ffd640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f640c06aef0 0x7f640c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.075+0000 7f6422ffd640 1 -- 172.21.15.79:0/1357092408 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f64180aa980 con 0x7f6424101da0 2024-06-01T00:51:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.075+0000 7f6429c66640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f640c06aef0 0x7f640c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.075+0000 7f6429c66640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f640c06aef0 0x7f640c06d3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f641400a810 tx=0x7f6414008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:49.080 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.079+0000 7f6422ffd640 1 -- 172.21.15.79:0/1357092408 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6418075fe0 con 0x7f6424101da0 2024-06-01T00:51:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.257+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6424104af0 con 0x7f6424101da0 2024-06-01T00:51:49.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.257+0000 7f6422ffd640 1 -- 172.21.15.79:0/1357092408 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6418079600 con 0x7f6424101da0 2024-06-01T00:51:49.258 INFO:teuthology.orchestra.run.smithi079.stdout:171798691852 2024-06-01T00:51:49.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f640c06aef0 msgr2=0x7f640c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:49.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f640c06aef0 0x7f640c06d3b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f641400a810 tx=0x7f6414008040 comp rx=0 tx=0).stop 2024-06-01T00:51:49.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 msgr2=0x7f64241968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:49.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f64241968c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f641802aa50 tx=0x7f64180055a0 comp rx=0 tx=0).stop 2024-06-01T00:51:49.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 shutdown_connections 2024-06-01T00:51:49.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f640c06aef0 0x7f640c06d3b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:49.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6424101da0 0x7f64241968c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 --2- 172.21.15.79:0/1357092408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6424101480 0x7f6424196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 >> 172.21.15.79:0/1357092408 conn(0x7f64240fac70 msgr2=0x7f64241026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 shutdown_connections 2024-06-01T00:51:49.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:49.259+0000 7f642beca640 1 -- 172.21.15.79:0/1357092408 wait complete. 2024-06-01T00:51:49.555 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:49 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1357092408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-01T00:51:49.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:49 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1357092408' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-01T00:51:50.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:50 smithi079 ceph-mon[29356]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:50.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:50.688 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691850 got 171798691852 for osd.5 2024-06-01T00:51:50.688 DEBUG:teuthology.parallel:result is None 2024-06-01T00:51:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:50 smithi160 ceph-mon[36639]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:50.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 -- 172.21.15.79:0/431567788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac80fe9f0 msgr2=0x7f8ac80fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/431567788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac80fe9f0 0x7f8ac80fee30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009880 tx=0x7f8abc02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 -- 172.21.15.79:0/431567788 shutdown_connections 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/431567788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac80fe9f0 0x7f8ac80fee30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/431567788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8ac81062f0 0x7f8ac81066d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 -- 172.21.15.79:0/431567788 >> 172.21.15.79:0/431567788 conn(0x7f8ac8074430 msgr2=0x7f8ac8074840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 -- 172.21.15.79:0/431567788 shutdown_connections 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.258+0000 7f8acf0e3640 1 -- 172.21.15.79:0/431567788 wait complete. 2024-06-01T00:51:51.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.259+0000 7f8acf0e3640 1 Processor -- start 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.259+0000 7f8acf0e3640 1 -- start start 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.259+0000 7f8acf0e3640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8ac80fe9f0 0x7f8ac8196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.259+0000 7f8acf0e3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac81062f0 0x7f8ac8196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.259+0000 7f8acf0e3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8196ef0 con 0x7f8ac81062f0 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.259+0000 7f8acf0e3640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac819ac10 con 0x7f8ac80fe9f0 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac81062f0 0x7f8ac8196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac7fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac81062f0 0x7f8ac8196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43130/0 (socket says 172.21.15.79:43130) 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac7fff640 1 -- 172.21.15.79:0/406044611 learned_addr learned my addr 172.21.15.79:0/406044611 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8acce7f640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8ac80fe9f0 0x7f8ac8196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac7fff640 1 -- 172.21.15.79:0/406044611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8ac80fe9f0 msgr2=0x7f8ac8196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac7fff640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8ac80fe9f0 0x7f8ac8196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac7fff640 1 -- 172.21.15.79:0/406044611 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc009560 con 0x7f8ac81062f0 2024-06-01T00:51:51.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8acce7f640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8ac80fe9f0 0x7f8ac8196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:51:51.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac7fff640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac81062f0 0x7f8ac8196860 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8abc02a9c0 tx=0x7f8abc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:51.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac5ffb640 1 -- 172.21.15.79:0/406044611 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8abc004aa0 con 0x7f8ac81062f0 2024-06-01T00:51:51.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac5ffb640 1 -- 172.21.15.79:0/406044611 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8abc004c00 con 0x7f8ac81062f0 2024-06-01T00:51:51.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac819ae90 con 0x7f8ac81062f0 2024-06-01T00:51:51.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8ac5ffb640 1 -- 172.21.15.79:0/406044611 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8abc003ab0 con 0x7f8ac81062f0 2024-06-01T00:51:51.264 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.260+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac819b380 con 0x7f8ac81062f0 2024-06-01T00:51:51.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.262+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac8068410 con 0x7f8ac81062f0 2024-06-01T00:51:51.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.262+0000 7f8ac5ffb640 1 -- 172.21.15.79:0/406044611 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f8abc003ce0 con 0x7f8ac81062f0 2024-06-01T00:51:51.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.263+0000 7f8ac5ffb640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8ab006aef0 0x7f8ab006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:51.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.263+0000 7f8ac5ffb640 1 -- 172.21.15.79:0/406044611 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8abc0aace0 con 0x7f8ac81062f0 2024-06-01T00:51:51.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.263+0000 7f8acce7f640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8ab006aef0 0x7f8ab006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:51.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.263+0000 7f8acce7f640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8ab006aef0 0x7f8ab006d3b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8009930 tx=0x7f8ab8008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:51.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.267+0000 7f8ac5ffb640 1 -- 172.21.15.79:0/406044611 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8abc0762c0 con 0x7f8ac81062f0 2024-06-01T00:51:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.444+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8ac804e210 con 0x7f8ac81062f0 2024-06-01T00:51:51.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.444+0000 7f8ac5ffb640 1 -- 172.21.15.79:0/406044611 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8abc0798e0 con 0x7f8ac81062f0 2024-06-01T00:51:51.445 INFO:teuthology.orchestra.run.smithi079.stdout:150323855377 2024-06-01T00:51:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8ab006aef0 msgr2=0x7f8ab006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8ab006aef0 0x7f8ab006d3b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8009930 tx=0x7f8ab8008040 comp rx=0 tx=0).stop 2024-06-01T00:51:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac81062f0 msgr2=0x7f8ac8196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac81062f0 0x7f8ac8196860 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8abc02a9c0 tx=0x7f8abc0055a0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 shutdown_connections 2024-06-01T00:51:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8ab006aef0 0x7f8ab006d3b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8ac81062f0 0x7f8ac8196860 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 --2- 172.21.15.79:0/406044611 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8ac80fe9f0 0x7f8ac8196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 >> 172.21.15.79:0/406044611 conn(0x7f8ac8074430 msgr2=0x7f8ac80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 shutdown_connections 2024-06-01T00:51:51.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.446+0000 7f8acf0e3640 1 -- 172.21.15.79:0/406044611 wait complete. 2024-06-01T00:51:51.532 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:51 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/406044611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-01T00:51:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 -- 172.21.15.79:0/1996519438 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 msgr2=0x7f76800ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 --2- 172.21.15.79:0/1996519438 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f76800ff780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7670009880 tx=0x7f767002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 -- 172.21.15.79:0/1996519438 shutdown_connections 2024-06-01T00:51:51.614 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 --2- 172.21.15.79:0/1996519438 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f76800ffd50 0x7f7680103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 --2- 172.21.15.79:0/1996519438 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f76800ff780 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 -- 172.21.15.79:0/1996519438 >> 172.21.15.79:0/1996519438 conn(0x7f76800fac50 msgr2=0x7f76800fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 -- 172.21.15.79:0/1996519438 shutdown_connections 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.610+0000 7f768669e640 1 -- 172.21.15.79:0/1996519438 wait complete. 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f768669e640 1 Processor -- start 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f768669e640 1 -- start start 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f768669e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f7680196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f768669e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f76800ffd50 0x7f76801967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f768669e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7680196e40 con 0x7f76800ff3a0 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f768669e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768019ab60 con 0x7f76800ffd50 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f767ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f7680196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f767ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f7680196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43160/0 (socket says 172.21.15.79:43160) 2024-06-01T00:51:51.615 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f767ffff640 1 -- 172.21.15.79:0/4143353688 learned_addr learned my addr 172.21.15.79:0/4143353688 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.611+0000 7f767f7fe640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f76800ffd50 0x7f76801967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f767ffff640 1 -- 172.21.15.79:0/4143353688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f76800ffd50 msgr2=0x7f76801967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f767ffff640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f76800ffd50 0x7f76801967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f767ffff640 1 -- 172.21.15.79:0/4143353688 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7670009560 con 0x7f76800ff3a0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f767ffff640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f7680196270 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f767002a9c0 tx=0x7f76700055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f767d7fa640 1 -- 172.21.15.79:0/4143353688 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7670004aa0 con 0x7f76800ff3a0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f767d7fa640 1 -- 172.21.15.79:0/4143353688 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7670004c00 con 0x7f76800ff3a0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768019ade0 con 0x7f76800ff3a0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f767d7fa640 1 -- 172.21.15.79:0/4143353688 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7670003980 con 0x7f76800ff3a0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.612+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768019b2d0 con 0x7f76800ff3a0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.613+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7680100880 con 0x7f76800ff3a0 2024-06-01T00:51:51.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.613+0000 7f767d7fa640 1 -- 172.21.15.79:0/4143353688 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7670003bf0 con 0x7f76800ff3a0 2024-06-01T00:51:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.614+0000 7f767d7fa640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f766806aef0 0x7f766806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.614+0000 7f767d7fa640 1 -- 172.21.15.79:0/4143353688 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f76700aabd0 con 0x7f76800ff3a0 2024-06-01T00:51:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.614+0000 7f767f7fe640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f766806aef0 0x7f766806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:51.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.614+0000 7f767f7fe640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f766806aef0 0x7f766806d3b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76801977d0 tx=0x7f767400b390 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:51.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.618+0000 7f767d7fa640 1 -- 172.21.15.79:0/4143353688 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7670076260 con 0x7f76800ff3a0 2024-06-01T00:51:51.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.791+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f768004e210 con 0x7f76800ff3a0 2024-06-01T00:51:51.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.791+0000 7f767d7fa640 1 -- 172.21.15.79:0/4143353688 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7670079880 con 0x7f76800ff3a0 2024-06-01T00:51:51.793 INFO:teuthology.orchestra.run.smithi079.stdout:60129542177 2024-06-01T00:51:51.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f766806aef0 msgr2=0x7f766806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f766806aef0 0x7f766806d3b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76801977d0 tx=0x7f767400b390 comp rx=0 tx=0).stop 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 msgr2=0x7f7680196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f7680196270 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f767002a9c0 tx=0x7f76700055a0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 shutdown_connections 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f766806aef0 0x7f766806d3b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f76800ffd50 0x7f76801967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 --2- 172.21.15.79:0/4143353688 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76800ff3a0 0x7f7680196270 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.793+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 >> 172.21.15.79:0/4143353688 conn(0x7f76800fac50 msgr2=0x7f7680068a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.794+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 shutdown_connections 2024-06-01T00:51:51.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:51.794+0000 7f768669e640 1 -- 172.21.15.79:0/4143353688 wait complete. 2024-06-01T00:51:51.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:51 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/406044611' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-01T00:51:52.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542177 for osd.1 2024-06-01T00:51:52.474 DEBUG:teuthology.parallel:result is None 2024-06-01T00:51:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:52 smithi160 ceph-mon[36639]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:52 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4143353688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-01T00:51:52.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:52 smithi079 ceph-mon[29356]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:52.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:52 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4143353688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-01T00:51:52.901 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855375 got 150323855377 for osd.4 2024-06-01T00:51:52.902 DEBUG:teuthology.parallel:result is None 2024-06-01T00:51:52.902 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-01T00:51:52.902 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-01T00:51:53.114 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:54.553 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:54 smithi079 ceph-mon[29356]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 -- 172.21.15.79:0/2740576345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3781013e0 msgr2=0x7ff3781017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 --2- 172.21.15.79:0/2740576345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3781013e0 0x7ff3781017c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff360009880 tx=0x7ff36002a530 comp rx=0 tx=0).stop 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 -- 172.21.15.79:0/2740576345 shutdown_connections 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 --2- 172.21.15.79:0/2740576345 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff378101d00 0x7ff37810a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 --2- 172.21.15.79:0/2740576345 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3781013e0 0x7ff3781017c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 -- 172.21.15.79:0/2740576345 >> 172.21.15.79:0/2740576345 conn(0x7ff3780fac10 msgr2=0x7ff3780fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 -- 172.21.15.79:0/2740576345 shutdown_connections 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 -- 172.21.15.79:0/2740576345 wait complete. 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.751+0000 7ff37f235640 1 Processor -- start 2024-06-01T00:51:54.764 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff37f235640 1 -- start start 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff37f235640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3781013e0 0x7ff3781961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff37f235640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff378101d00 0x7ff378196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff37f235640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff378196d20 con 0x7ff378101d00 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff37f235640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff378196e90 con 0x7ff3781013e0 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff377fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff378101d00 0x7ff378196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff377fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff378101d00 0x7ff378196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57352/0 (socket says 172.21.15.79:57352) 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff377fff640 1 -- 172.21.15.79:0/2246668528 learned_addr learned my addr 172.21.15.79:0/2246668528 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff37cfd1640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3781013e0 0x7ff3781961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff377fff640 1 -- 172.21.15.79:0/2246668528 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3781013e0 msgr2=0x7ff3781961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff377fff640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3781013e0 0x7ff3781961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff377fff640 1 -- 172.21.15.79:0/2246668528 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff360009560 con 0x7ff378101d00 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff377fff640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff378101d00 0x7ff378196720 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff36800c8f0 tx=0x7ff36800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:54.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff375ffb640 1 -- 172.21.15.79:0/2246668528 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3680041f0 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff375ffb640 1 -- 172.21.15.79:0/2246668528 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff368007e70 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.752+0000 7ff375ffb640 1 -- 172.21.15.79:0/2246668528 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff36800f450 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.753+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37819ac20 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.753+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37819b170 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.754+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff378068fe0 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.754+0000 7ff375ffb640 1 -- 172.21.15.79:0/2246668528 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ff368004350 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.754+0000 7ff375ffb640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff36406aef0 0x7ff36406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.755+0000 7ff37cfd1640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff36406aef0 0x7ff36406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.755+0000 7ff375ffb640 1 -- 172.21.15.79:0/2246668528 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff368013070 con 0x7ff378101d00 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.755+0000 7ff37cfd1640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff36406aef0 0x7ff36406d3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff360005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:54.766 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.759+0000 7ff375ffb640 1 -- 172.21.15.79:0/2246668528 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff368055140 con 0x7ff378101d00 2024-06-01T00:51:54.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:54 smithi160 ceph-mon[36639]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:54.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.928+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff37810d4b0 con 0x7ff36406aef0 2024-06-01T00:51:54.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.930+0000 7ff375ffb640 1 -- 172.21.15.79:0/2246668528 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28173 (secure 0 0 0) 0x7ff37810d4b0 con 0x7ff36406aef0 2024-06-01T00:51:54.931 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:54.932 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff36406aef0 msgr2=0x7ff36406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff36406aef0 0x7ff36406d3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff360005a20 comp rx=0 tx=0).stop 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff378101d00 msgr2=0x7ff378196720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff378101d00 0x7ff378196720 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff36800c8f0 tx=0x7ff36800cdb0 comp rx=0 tx=0).stop 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 shutdown_connections 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff36406aef0 0x7ff36406d3b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff378101d00 0x7ff378196720 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 --2- 172.21.15.79:0/2246668528 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3781013e0 0x7ff3781961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:54.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 >> 172.21.15.79:0/2246668528 conn(0x7ff3780fac10 msgr2=0x7ff3781026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:54.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 shutdown_connections 2024-06-01T00:51:54.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:54.932+0000 7ff37f235640 1 -- 172.21.15.79:0/2246668528 wait complete. 2024-06-01T00:51:55.640 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-06-01T00:51:53.321590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":182448,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":562288464,"statfs":{"total":575970213888,"available":575783387136,"internally_reserved":0,"allocated":4018176,"data_stored":2628132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":182704806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002069"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-01T00:51:03.644969+0000","last_change":"2024-06-01T00:50:37.392887+0000","last_active":"2024-06-01T00:51:03.644969+0000","last_peered":"2024-06-01T00:51:03.644969+0000","last_clean":"2024-06-01T00:51:03.644969+0000","last_became_active":"2024-06-01T00:50:37.392624+0000","last_became_peered":"2024-06-01T00:50:37.392624+0000","last_unstale":"2024-06-01T00:51:03.644969+0000","last_undegraded":"2024-06-01T00:51:03.644969+0000","last_fullsized":"2024-06-01T00:51:03.644969+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-01T00:49:52.229241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-01T00:49:52.229241+0000","last_clean_scrub_stamp":"2024-06-01T00:49:52.229241+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-02T09:10:05.566542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31484,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713668,"statfs":{"total":95995035648,"available":95962796032,"internally_reserved":0,"allocated":372736,"data_stored":142838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.502}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.434}]},{"osd":1,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.502},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Jun 1 00:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31484,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713668,"statfs":{"total":95995035648,"available":95962796032,"internally_reserved":0,"allocated":372736,"data_stored":142838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.24399999999999999}]},{"osd":1,"last update":"Sat Jun 1 00:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.27300000000000002}]},{"osd":3,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.499},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Sat Jun 1 00:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":966656,"data_stored":733206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:51:27 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41499999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.22},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.311,"15min":0.309},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"last":0.23100000000000001}]},{"osd":2,"last update":"Sat Jun 1 00:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.247},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"last":0.20499999999999999}]},{"osd":3,"last update":"Sat Jun 1 00:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.33700000000000002}]},{"osd":4,"last update":"Sat Jun 1 00:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":966656,"data_stored":733206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 1 00:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.28799999999999998},"min":{"1min":0.24199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Jun 1 00:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.435,"5min":0.435,"15min":0.435},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.254}]},{"osd":3,"last update":"Sat Jun 1 00:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Sat Jun 1 00:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.626},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-01T00:51:55.642 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-01T00:51:55.846 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:56.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:56 smithi079 ceph-mon[29356]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T00:51:56.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:56 smithi079 ceph-mon[29356]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:56.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:56 smithi160 ceph-mon[36639]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T00:51:56.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:56 smithi160 ceph-mon[36639]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:57.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 -- 172.21.15.79:0/1815506260 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101d20 msgr2=0x7fa26010a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:57.436 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 --2- 172.21.15.79:0/1815506260 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101d20 0x7fa26010a210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa248009880 tx=0x7fa24802a530 comp rx=0 tx=0).stop 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 -- 172.21.15.79:0/1815506260 shutdown_connections 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 --2- 172.21.15.79:0/1815506260 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101d20 0x7fa26010a210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 --2- 172.21.15.79:0/1815506260 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa260101400 0x7fa2601017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 -- 172.21.15.79:0/1815506260 >> 172.21.15.79:0/1815506260 conn(0x7fa2600fac50 msgr2=0x7fa2600fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 -- 172.21.15.79:0/1815506260 shutdown_connections 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.414+0000 7fa266a30640 1 -- 172.21.15.79:0/1815506260 wait complete. 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa266a30640 1 Processor -- start 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa266a30640 1 -- start start 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa266a30640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101400 0x7fa260196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa266a30640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa260101d20 0x7fa260196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa266a30640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa260196fd0 con 0x7fa260101400 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa266a30640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26019acf0 con 0x7fa260101d20 2024-06-01T00:51:57.437 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101400 0x7fa260196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa25ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101400 0x7fa260196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57372/0 (socket says 172.21.15.79:57372) 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.415+0000 7fa25ffff640 1 -- 172.21.15.79:0/2722115493 learned_addr learned my addr 172.21.15.79:0/2722115493 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa25ffff640 1 -- 172.21.15.79:0/2722115493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa260101d20 msgr2=0x7fa260196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa25ffff640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa260101d20 0x7fa260196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa25ffff640 1 -- 172.21.15.79:0/2722115493 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248009560 con 0x7fa260101400 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa25ffff640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101400 0x7fa260196400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa25000b420 tx=0x7fa25000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa25d7fa640 1 -- 172.21.15.79:0/2722115493 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa250011800 con 0x7fa260101400 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa25d7fa640 1 -- 172.21.15.79:0/2722115493 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa250011e20 con 0x7fa260101400 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26019afd0 con 0x7fa260101400 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa25d7fa640 1 -- 172.21.15.79:0/2722115493 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa25000f2e0 con 0x7fa260101400 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.416+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26019b520 con 0x7fa260101400 2024-06-01T00:51:57.438 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.417+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa260069080 con 0x7fa260101400 2024-06-01T00:51:57.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.417+0000 7fa25d7fa640 1 -- 172.21.15.79:0/2722115493 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fa250010380 con 0x7fa260101400 2024-06-01T00:51:57.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.418+0000 7fa25d7fa640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa24c06aef0 0x7fa24c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:51:57.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.418+0000 7fa25d7fa640 1 -- 172.21.15.79:0/2722115493 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa250089570 con 0x7fa260101400 2024-06-01T00:51:57.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.418+0000 7fa25f7fe640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa24c06aef0 0x7fa24c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:51:57.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.418+0000 7fa25f7fe640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa24c06aef0 0x7fa24c06d3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa260197960 tx=0x7fa248000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:51:57.439 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.422+0000 7fa25d7fa640 1 -- 172.21.15.79:0/2722115493 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa250054b50 con 0x7fa260101400 2024-06-01T00:51:57.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.590+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 --> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa26010d550 con 0x7fa24c06aef0 2024-06-01T00:51:57.592 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.592+0000 7fa25d7fa640 1 -- 172.21.15.79:0/2722115493 <== mgr.14215 v2:172.21.15.79:6800/187237089 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28187 (secure 0 0 0) 0x7fa26010d550 con 0x7fa24c06aef0 2024-06-01T00:51:57.593 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:51:57.593 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-06-01T00:51:57.595 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.594+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa24c06aef0 msgr2=0x7fa24c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.594+0000 7fa266a30640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa24c06aef0 0x7fa24c06d3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa260197960 tx=0x7fa248000bc0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.594+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101400 msgr2=0x7fa260196400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.594+0000 7fa266a30640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101400 0x7fa260196400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa25000b420 tx=0x7fa25000b8e0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.595+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 shutdown_connections 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.595+0000 7fa266a30640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa24c06aef0 0x7fa24c06d3b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.595+0000 7fa266a30640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa260101d20 0x7fa260196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.595+0000 7fa266a30640 1 --2- 172.21.15.79:0/2722115493 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa260101400 0x7fa260196400 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.595+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 >> 172.21.15.79:0/2722115493 conn(0x7fa2600fac50 msgr2=0x7fa2601026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.595+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 shutdown_connections 2024-06-01T00:51:57.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:51:57.595+0000 7fa266a30640 1 -- 172.21.15.79:0/2722115493 wait complete. 2024-06-01T00:51:58.219 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-06-01T00:51:57.322360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":182448,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":178422,"kb_avail":562288464,"statfs":{"total":575970213888,"available":575783387136,"internally_reserved":0,"allocated":4018176,"data_stored":2628132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":182704806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002029"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-01T00:51:03.644969+0000","last_change":"2024-06-01T00:50:37.392887+0000","last_active":"2024-06-01T00:51:03.644969+0000","last_peered":"2024-06-01T00:51:03.644969+0000","last_clean":"2024-06-01T00:51:03.644969+0000","last_became_active":"2024-06-01T00:50:37.392624+0000","last_became_peered":"2024-06-01T00:50:37.392624+0000","last_unstale":"2024-06-01T00:51:03.644969+0000","last_undegraded":"2024-06-01T00:51:03.644969+0000","last_fullsized":"2024-06-01T00:51:03.644969+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-01T00:49:52.229241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-01T00:49:52.229241+0000","last_clean_scrub_stamp":"2024-06-01T00:49:52.229241+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-02T09:10:05.566542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31484,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713668,"statfs":{"total":95995035648,"available":95962796032,"internally_reserved":0,"allocated":372736,"data_stored":142838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.57199999999999995}]},{"osd":2,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Jun 1 00:51:40 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.502}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.434}]},{"osd":1,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.502},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Jun 1 00:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31484,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713668,"statfs":{"total":95995035648,"available":95962796032,"internally_reserved":0,"allocated":372736,"data_stored":142838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.28899999999999998,"5min":0.311,"15min":0.315},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.36299999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.318},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.32800000000000001,"15min":0.33300000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.434,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.27600000000000002}]},{"osd":1,"last update":"Sat Jun 1 00:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.34699999999999998,"15min":0.35399999999999998},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.45500000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.35299999999999998,"15min":0.36099999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.41299999999999998,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Jun 1 00:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.57199999999999995}]},{"osd":4,"last update":"Sat Jun 1 00:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":966656,"data_stored":733206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 00:51:27 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.41499999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"last":0.22},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.311,"15min":0.309},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"last":0.23100000000000001}]},{"osd":2,"last update":"Sat Jun 1 00:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.247},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"last":0.20499999999999999}]},{"osd":3,"last update":"Sat Jun 1 00:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.33700000000000002}]},{"osd":4,"last update":"Sat Jun 1 00:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":966656,"data_stored":733206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 1 00:51:28 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.29499999999999998,"15min":0.28799999999999998},"min":{"1min":0.24199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.29599999999999999,"15min":0.29099999999999998},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Jun 1 00:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.435,"5min":0.435,"15min":0.435},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"last":0.254}]},{"osd":3,"last update":"Sat Jun 1 00:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.58999999999999997}]},{"osd":4,"last update":"Sat Jun 1 00:51:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.626},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-01T00:51:58.220 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-01T00:51:58.221 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-01T00:51:58.221 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-01T00:51:58.221 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:51:58.423 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:51:58.783 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:58 smithi079 ceph-mon[29356]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:58.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:58 smithi160 ceph-mon[36639]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:51:59.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:51:59 smithi079 ceph-mon[29356]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T00:51:59.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:51:59 smithi160 ceph-mon[36639]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 -- 172.21.15.79:0/3384101638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ff3b0 msgr2=0x7f10f40ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 --2- 172.21.15.79:0/3384101638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ff3b0 0x7f10f40ff790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009c20 tx=0x7f10ec02a970 comp rx=0 tx=0).stop 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 -- 172.21.15.79:0/3384101638 shutdown_connections 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 --2- 172.21.15.79:0/3384101638 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10f40ffd60 0x7f10f4103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 --2- 172.21.15.79:0/3384101638 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ff3b0 0x7f10f40ff790 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 -- 172.21.15.79:0/3384101638 >> 172.21.15.79:0/3384101638 conn(0x7f10f40fac60 msgr2=0x7f10f40fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 -- 172.21.15.79:0/3384101638 shutdown_connections 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 -- 172.21.15.79:0/3384101638 wait complete. 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 Processor -- start 2024-06-01T00:52:00.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.148+0000 7f10fba7e640 1 -- start start 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10fba7e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10f40ff3b0 0x7f10f4195030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10fba7e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ffd60 0x7f10f4195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10fba7e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f4195ab0 con 0x7f10f40ffd60 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10fba7e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f4195c20 con 0x7f10f40ff3b0 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f9019640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ffd60 0x7f10f4195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f9019640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ffd60 0x7f10f4195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57400/0 (socket says 172.21.15.79:57400) 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f9019640 1 -- 172.21.15.79:0/1072256653 learned_addr learned my addr 172.21.15.79:0/1072256653 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f981a640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10f40ff3b0 0x7f10f4195030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f9019640 1 -- 172.21.15.79:0/1072256653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10f40ff3b0 msgr2=0x7f10f4195030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f9019640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10f40ff3b0 0x7f10f4195030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:00.151 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f9019640 1 -- 172.21.15.79:0/1072256653 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec009900 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f9019640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ffd60 0x7f10f4195570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b3d0 tx=0x7f10e800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f2ffd640 1 -- 172.21.15.79:0/1072256653 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e8011800 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f2ffd640 1 -- 172.21.15.79:0/1072256653 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10e8011e20 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.149+0000 7f10f2ffd640 1 -- 172.21.15.79:0/1072256653 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e800f2e0 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.150+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f41a26a0 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.150+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f41a2a00 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.150+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f4100890 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.151+0000 7f10f2ffd640 1 -- 172.21.15.79:0/1072256653 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f10e8010380 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.151+0000 7f10f2ffd640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f10dc06aef0 0x7f10dc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.152+0000 7f10f2ffd640 1 -- 172.21.15.79:0/1072256653 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f10e80898c0 con 0x7f10f40ffd60 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.152+0000 7f10f981a640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f10dc06aef0 0x7f10dc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:00.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.152+0000 7f10f981a640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f10dc06aef0 0x7f10dc06d3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0098d0 tx=0x7f10ec009800 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:00.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.156+0000 7f10f2ffd640 1 -- 172.21.15.79:0/1072256653 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f10e8054ea0 con 0x7f10f40ffd60 2024-06-01T00:52:00.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.366+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f10f410c460 con 0x7f10f40ffd60 2024-06-01T00:52:00.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.367+0000 7f10f2ffd640 1 -- 172.21.15.79:0/1072256653 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f10e80584c0 con 0x7f10f40ffd60 2024-06-01T00:52:00.368 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:00.368 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f10dc06aef0 msgr2=0x7f10dc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f10dc06aef0 0x7f10dc06d3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0098d0 tx=0x7f10ec009800 comp rx=0 tx=0).stop 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ffd60 msgr2=0x7f10f4195570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ffd60 0x7f10f4195570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f10e800b3d0 tx=0x7f10e800b890 comp rx=0 tx=0).stop 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 shutdown_connections 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f10dc06aef0 0x7f10dc06d3b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10f40ffd60 0x7f10f4195570 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 --2- 172.21.15.79:0/1072256653 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f10f40ff3b0 0x7f10f4195030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:00.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 >> 172.21.15.79:0/1072256653 conn(0x7f10f40fac60 msgr2=0x7f10f4102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:00.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 shutdown_connections 2024-06-01T00:52:00.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:00.369+0000 7f10fba7e640 1 -- 172.21.15.79:0/1072256653 wait complete. 2024-06-01T00:52:00.711 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:00 smithi079 ceph-mon[29356]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:00.712 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:00 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1072256653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:00.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:00 smithi160 ceph-mon[36639]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:00.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:00 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1072256653' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:02.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:02 smithi160 ceph-mon[36639]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:02.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:02 smithi079 ceph-mon[29356]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:04.094 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:04.312 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:04.697 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:04 smithi079 ceph-mon[29356]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:04.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:04 smithi160 ceph-mon[36639]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:05.754 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:52:05.754 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:05.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:52:05.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:06.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.109+0000 7fadbc314640 1 -- 172.21.15.79:0/2932523675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 msgr2=0x7fadb410a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:06.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.109+0000 7fadbc314640 1 --2- 172.21.15.79:0/2932523675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb410a1f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fada40098b0 tx=0x7fada402a4d0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 -- 172.21.15.79:0/2932523675 shutdown_connections 2024-06-01T00:52:06.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 --2- 172.21.15.79:0/2932523675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb410a1f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 --2- 172.21.15.79:0/2932523675 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadb4101350 0x7fadb4101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 -- 172.21.15.79:0/2932523675 >> 172.21.15.79:0/2932523675 conn(0x7fadb40fac30 msgr2=0x7fadb40fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 -- 172.21.15.79:0/2932523675 shutdown_connections 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 -- 172.21.15.79:0/2932523675 wait complete. 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 Processor -- start 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.110+0000 7fadbc314640 1 -- start start 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadbc314640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadb4101350 0x7fadb4196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadbc314640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb4196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadbc314640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4196e20 con 0x7fadb4101d00 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadbc314640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb419ab40 con 0x7fadb4101350 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadb98af640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb4196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadb98af640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb4196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42306/0 (socket says 172.21.15.79:42306) 2024-06-01T00:52:06.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadb98af640 1 -- 172.21.15.79:0/2680264310 learned_addr learned my addr 172.21.15.79:0/2680264310 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadba0b0640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadb4101350 0x7fadb4196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadb98af640 1 -- 172.21.15.79:0/2680264310 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadb4101350 msgr2=0x7fadb4196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadb98af640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadb4101350 0x7fadb4196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadb98af640 1 -- 172.21.15.79:0/2680264310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada4009560 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadb98af640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb4196790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fada4006010 tx=0x7fada40055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.111+0000 7fadab7fe640 1 -- 172.21.15.79:0/2680264310 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada40048d0 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.112+0000 7fadab7fe640 1 -- 172.21.15.79:0/2680264310 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fada4031640 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.112+0000 7fadab7fe640 1 -- 172.21.15.79:0/2680264310 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada4003820 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.112+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb419adc0 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.112+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb419b260 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.113+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb4068da0 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.113+0000 7fadab7fe640 1 -- 172.21.15.79:0/2680264310 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fada40317b0 con 0x7fadb4101d00 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.114+0000 7fadab7fe640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fada006aef0 0x7fada006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:06.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.114+0000 7fadba0b0640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fada006aef0 0x7fada006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:06.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.114+0000 7fadab7fe640 1 -- 172.21.15.79:0/2680264310 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fada40aab70 con 0x7fadb4101d00 2024-06-01T00:52:06.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.114+0000 7fadba0b0640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fada006aef0 0x7fada006d3b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009bd0 tx=0x7fadb0005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:06.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.118+0000 7fadab7fe640 1 -- 172.21.15.79:0/2680264310 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fada4076200 con 0x7fadb4101d00 2024-06-01T00:52:06.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.335+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fadb404e210 con 0x7fadb4101d00 2024-06-01T00:52:06.336 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.335+0000 7fadab7fe640 1 -- 172.21.15.79:0/2680264310 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fada4079820 con 0x7fadb4101d00 2024-06-01T00:52:06.337 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:06.337 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:06.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fada006aef0 msgr2=0x7fada006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:06.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fada006aef0 0x7fada006d3b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009bd0 tx=0x7fadb0005f90 comp rx=0 tx=0).stop 2024-06-01T00:52:06.339 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 msgr2=0x7fadb4196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb4196790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fada4006010 tx=0x7fada40055a0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 shutdown_connections 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fada006aef0 0x7fada006d3b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fadb4101d00 0x7fadb4196790 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 --2- 172.21.15.79:0/2680264310 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fadb4101350 0x7fadb4196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 >> 172.21.15.79:0/2680264310 conn(0x7fadb40fac30 msgr2=0x7fadb4102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 shutdown_connections 2024-06-01T00:52:06.340 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:06.338+0000 7fadbc314640 1 -- 172.21.15.79:0/2680264310 wait complete. 2024-06-01T00:52:06.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:06 smithi079 ceph-mon[29356]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:06.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:52:06.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:52:06.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:52:06.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:06 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2680264310' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:06 smithi160 ceph-mon[36639]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:52:06.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:52:06.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:52:06.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:06 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2680264310' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:08.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:08 smithi160 ceph-mon[36639]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:08.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:08 smithi079 ceph-mon[29356]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:09.964 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:10.184 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:10.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:10 smithi079 ceph-mon[29356]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:10.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:10 smithi160 ceph-mon[36639]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:11.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 -- 172.21.15.79:0/4249850609 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 msgr2=0x7f4b6c103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:11.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 --2- 172.21.15.79:0/4249850609 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c103d60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009880 tx=0x7f4b5c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:52:11.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 -- 172.21.15.79:0/4249850609 shutdown_connections 2024-06-01T00:52:11.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 --2- 172.21.15.79:0/4249850609 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c103d60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:11.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 --2- 172.21.15.79:0/4249850609 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c0ff360 0x7f4b6c0ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:11.808 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 -- 172.21.15.79:0/4249850609 >> 172.21.15.79:0/4249850609 conn(0x7f4b6c075e10 msgr2=0x7f4b6c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:11.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 -- 172.21.15.79:0/4249850609 shutdown_connections 2024-06-01T00:52:11.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 -- 172.21.15.79:0/4249850609 wait complete. 2024-06-01T00:52:11.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 Processor -- start 2024-06-01T00:52:11.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.793+0000 7f4b72df8640 1 -- start start 2024-06-01T00:52:11.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b72df8640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c0ff360 0x7f4b6c197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:11.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b72df8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:11.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b72df8640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c197b10 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b72df8640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c197c80 con 0x7f4b6c0ff360 2024-06-01T00:52:11.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b63fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:11.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b63fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c1975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42322/0 (socket says 172.21.15.79:42322) 2024-06-01T00:52:11.810 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b63fff640 1 -- 172.21.15.79:0/250377872 learned_addr learned my addr 172.21.15.79:0/250377872 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b70b94640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c0ff360 0x7f4b6c197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b63fff640 1 -- 172.21.15.79:0/250377872 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c0ff360 msgr2=0x7f4b6c197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b63fff640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c0ff360 0x7f4b6c197090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b63fff640 1 -- 172.21.15.79:0/250377872 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c009560 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b63fff640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c1975d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c02a9c0 tx=0x7f4b5c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b61ffb640 1 -- 172.21.15.79:0/250377872 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5c004aa0 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b61ffb640 1 -- 172.21.15.79:0/250377872 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b5c031470 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c197f00 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.794+0000 7f4b61ffb640 1 -- 172.21.15.79:0/250377872 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b5c0039e0 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.795+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c1a0430 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.796+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c100840 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.796+0000 7f4b61ffb640 1 -- 172.21.15.79:0/250377872 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f4b5c003be0 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.796+0000 7f4b61ffb640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b5406aef0 0x7f4b5406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:11.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.796+0000 7f4b61ffb640 1 -- 172.21.15.79:0/250377872 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4b5c0aaa30 con 0x7f4b6c0ffd10 2024-06-01T00:52:11.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.796+0000 7f4b70b94640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b5406aef0 0x7f4b5406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:11.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.797+0000 7f4b70b94640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b5406aef0 0x7f4b5406d3b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4b68005fd0 tx=0x7f4b6800b390 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:11.812 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:11.801+0000 7f4b61ffb640 1 -- 172.21.15.79:0/250377872 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4b5c0760c0 con 0x7f4b6c0ffd10 2024-06-01T00:52:12.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.006+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4b6c197f00 con 0x7f4b6c0ffd10 2024-06-01T00:52:12.009 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.007+0000 7f4b61ffb640 1 -- 172.21.15.79:0/250377872 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4b6c197f00 con 0x7f4b6c0ffd10 2024-06-01T00:52:12.009 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:12.009 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b5406aef0 msgr2=0x7f4b5406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b5406aef0 0x7f4b5406d3b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4b68005fd0 tx=0x7f4b6800b390 comp rx=0 tx=0).stop 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 msgr2=0x7f4b6c1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c1975d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c02a9c0 tx=0x7f4b5c0055a0 comp rx=0 tx=0).stop 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 shutdown_connections 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4b5406aef0 0x7f4b5406d3b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b6c0ffd10 0x7f4b6c1975d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 --2- 172.21.15.79:0/250377872 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4b6c0ff360 0x7f4b6c197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 >> 172.21.15.79:0/250377872 conn(0x7f4b6c075e10 msgr2=0x7f4b6c102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:12.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 shutdown_connections 2024-06-01T00:52:12.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:12.009+0000 7f4b72df8640 1 -- 172.21.15.79:0/250377872 wait complete. 2024-06-01T00:52:12.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:12 smithi079 ceph-mon[29356]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:12.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:12 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/250377872' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:12 smithi160 ceph-mon[36639]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:12 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/250377872' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:14.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:14 smithi160 ceph-mon[36639]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:14.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:14 smithi079 ceph-mon[29356]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:15.642 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:15.861 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:16.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:16 smithi079 ceph-mon[29356]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:16.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:16 smithi160 ceph-mon[36639]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:17.596 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 -- 172.21.15.79:0/1144987762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ffd10 msgr2=0x7f15a8103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:17.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 --2- 172.21.15.79:0/1144987762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ffd10 0x7f15a8103d60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f159c009880 tx=0x7f159c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 -- 172.21.15.79:0/1144987762 shutdown_connections 2024-06-01T00:52:17.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 --2- 172.21.15.79:0/1144987762 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ffd10 0x7f15a8103d60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 --2- 172.21.15.79:0/1144987762 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15a80ff360 0x7f15a80ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.597 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 -- 172.21.15.79:0/1144987762 >> 172.21.15.79:0/1144987762 conn(0x7f15a8075e10 msgr2=0x7f15a80fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:17.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 -- 172.21.15.79:0/1144987762 shutdown_connections 2024-06-01T00:52:17.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.577+0000 7f15add96640 1 -- 172.21.15.79:0/1144987762 wait complete. 2024-06-01T00:52:17.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15add96640 1 Processor -- start 2024-06-01T00:52:17.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15add96640 1 -- start start 2024-06-01T00:52:17.598 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15add96640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ff360 0x7f15a81952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:17.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15add96640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15a80ffd10 0x7f15a8195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:17.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15add96640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a8195e90 con 0x7f15a80ff360 2024-06-01T00:52:17.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15add96640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a8199bb0 con 0x7f15a80ffd10 2024-06-01T00:52:17.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ff360 0x7f15a81952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:17.599 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15a77fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ff360 0x7f15a81952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41744/0 (socket says 172.21.15.79:41744) 2024-06-01T00:52:17.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15a77fe640 1 -- 172.21.15.79:0/3792344108 learned_addr learned my addr 172.21.15.79:0/3792344108 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:17.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15a77fe640 1 -- 172.21.15.79:0/3792344108 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15a80ffd10 msgr2=0x7f15a8195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:17.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15a77fe640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15a80ffd10 0x7f15a8195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.578+0000 7f15a77fe640 1 -- 172.21.15.79:0/3792344108 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c009560 con 0x7f15a80ff360 2024-06-01T00:52:17.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.579+0000 7f15a77fe640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ff360 0x7f15a81952c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f159800b420 tx=0x7f159800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:17.600 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.579+0000 7f15a4ff9640 1 -- 172.21.15.79:0/3792344108 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1598011800 con 0x7f15a80ff360 2024-06-01T00:52:17.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.579+0000 7f15a4ff9640 1 -- 172.21.15.79:0/3792344108 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1598011e20 con 0x7f15a80ff360 2024-06-01T00:52:17.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.579+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a8199e90 con 0x7f15a80ff360 2024-06-01T00:52:17.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.579+0000 7f15a4ff9640 1 -- 172.21.15.79:0/3792344108 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f159800f2e0 con 0x7f15a80ff360 2024-06-01T00:52:17.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.579+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a819a3e0 con 0x7f15a80ff360 2024-06-01T00:52:17.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.580+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a8100840 con 0x7f15a80ff360 2024-06-01T00:52:17.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.580+0000 7f15a4ff9640 1 -- 172.21.15.79:0/3792344108 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f1598010380 con 0x7f15a80ff360 2024-06-01T00:52:17.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.581+0000 7f15a4ff9640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f159006aef0 0x7f159006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:17.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.581+0000 7f15a6ffd640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f159006aef0 0x7f159006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:17.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.581+0000 7f15a4ff9640 1 -- 172.21.15.79:0/3792344108 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f15980894d0 con 0x7f15a80ff360 2024-06-01T00:52:17.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.581+0000 7f15a6ffd640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f159006aef0 0x7f159006d3b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f15a8196820 tx=0x7f159c000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:17.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.585+0000 7f15a4ff9640 1 -- 172.21.15.79:0/3792344108 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1598054b30 con 0x7f15a80ff360 2024-06-01T00:52:17.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:17 smithi079 ceph-mon[29356]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:17.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.795+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f15a804e210 con 0x7f15a80ff360 2024-06-01T00:52:17.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.795+0000 7f15a4ff9640 1 -- 172.21.15.79:0/3792344108 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1598058150 con 0x7f15a80ff360 2024-06-01T00:52:17.796 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:17.796 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f159006aef0 msgr2=0x7f159006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f159006aef0 0x7f159006d3b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f15a8196820 tx=0x7f159c000bc0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ff360 msgr2=0x7f15a81952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ff360 0x7f15a81952c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f159800b420 tx=0x7f159800b8e0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 shutdown_connections 2024-06-01T00:52:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f159006aef0 0x7f159006d3b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f15a80ffd10 0x7f15a8195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 --2- 172.21.15.79:0/3792344108 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f15a80ff360 0x7f15a81952c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 >> 172.21.15.79:0/3792344108 conn(0x7f15a8075e10 msgr2=0x7f15a8102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 shutdown_connections 2024-06-01T00:52:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:17.797+0000 7f15add96640 1 -- 172.21.15.79:0/3792344108 wait complete. 2024-06-01T00:52:17.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:17 smithi160 ceph-mon[36639]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:18.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:18 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3792344108' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:18 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3792344108' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:19.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:19 smithi160 ceph-mon[36639]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:19.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:19 smithi079 ceph-mon[29356]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:20.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:21.372 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:21.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:21 smithi160 ceph-mon[36639]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:21.589 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:21.711 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:21 smithi079 ceph-mon[29356]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:23.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- 172.21.15.79:0/3608733780 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f041004b0 msgr2=0x7f2f04100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:23.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 --2- 172.21.15.79:0/3608733780 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f041004b0 0x7f2f04100890 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009880 tx=0x7f2f0002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- 172.21.15.79:0/3608733780 shutdown_connections 2024-06-01T00:52:23.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 --2- 172.21.15.79:0/3608733780 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f04100e60 0x7f2f04104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.175 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 --2- 172.21.15.79:0/3608733780 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f041004b0 0x7f2f04100890 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- 172.21.15.79:0/3608733780 >> 172.21.15.79:0/3608733780 conn(0x7f2f040fbd40 msgr2=0x7f2f040fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- 172.21.15.79:0/3608733780 shutdown_connections 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- 172.21.15.79:0/3608733780 wait complete. 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 Processor -- start 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- start start 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f041004b0 0x7f2f04196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04100e60 0x7f2f04196750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04196d50 con 0x7f2f04100e60 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.135+0000 7f2f0b096640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04196ec0 con 0x7f2f041004b0 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2efbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04100e60 0x7f2f04196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:23.176 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2efbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04100e60 0x7f2f04196750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55350/0 (socket says 172.21.15.79:55350) 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2efbfff640 1 -- 172.21.15.79:0/470127821 learned_addr learned my addr 172.21.15.79:0/470127821 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2f08e32640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f041004b0 0x7f2f04196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2efbfff640 1 -- 172.21.15.79:0/470127821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f041004b0 msgr2=0x7f2f04196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2efbfff640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f041004b0 0x7f2f04196210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2efbfff640 1 -- 172.21.15.79:0/470127821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00009560 con 0x7f2f04100e60 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2efbfff640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04100e60 0x7f2f04196750 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c8f0 tx=0x7f2ef400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2ef9ffb640 1 -- 172.21.15.79:0/470127821 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4004410 con 0x7f2f04100e60 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2ef9ffb640 1 -- 172.21.15.79:0/470127821 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ef4007c60 con 0x7f2f04100e60 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0419ac50 con 0x7f2f04100e60 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2ef9ffb640 1 -- 172.21.15.79:0/470127821 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef400f450 con 0x7f2f04100e60 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.136+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0419b1a0 con 0x7f2f04100e60 2024-06-01T00:52:23.177 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.137+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f04101990 con 0x7f2f04100e60 2024-06-01T00:52:23.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.138+0000 7f2ef9ffb640 1 -- 172.21.15.79:0/470127821 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2ef40077a0 con 0x7f2f04100e60 2024-06-01T00:52:23.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.138+0000 7f2ef9ffb640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2eec06aef0 0x7f2eec06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:23.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.138+0000 7f2ef9ffb640 1 -- 172.21.15.79:0/470127821 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2ef4089b10 con 0x7f2f04100e60 2024-06-01T00:52:23.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.138+0000 7f2f08e32640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2eec06aef0 0x7f2eec06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:23.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.139+0000 7f2f08e32640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2eec06aef0 0x7f2eec06d3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009230 tx=0x7f2f000058d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:23.178 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.142+0000 7f2ef9ffb640 1 -- 172.21.15.79:0/470127821 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2ef40551a0 con 0x7f2f04100e60 2024-06-01T00:52:23.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.349+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2f041087f0 con 0x7f2f04100e60 2024-06-01T00:52:23.351 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.350+0000 7f2ef9ffb640 1 -- 172.21.15.79:0/470127821 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2ef40587c0 con 0x7f2f04100e60 2024-06-01T00:52:23.351 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:23.351 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:23.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2eec06aef0 msgr2=0x7f2eec06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:23.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2eec06aef0 0x7f2eec06d3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009230 tx=0x7f2f000058d0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04100e60 msgr2=0x7f2f04196750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:23.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04100e60 0x7f2f04196750 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c8f0 tx=0x7f2ef400cdb0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 shutdown_connections 2024-06-01T00:52:23.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2eec06aef0 0x7f2eec06d3b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.353 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2f04100e60 0x7f2f04196750 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 --2- 172.21.15.79:0/470127821 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2f041004b0 0x7f2f04196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:23.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 >> 172.21.15.79:0/470127821 conn(0x7f2f040fbd40 msgr2=0x7f2f040fc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:23.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 shutdown_connections 2024-06-01T00:52:23.354 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:23.352+0000 7f2f0b096640 1 -- 172.21.15.79:0/470127821 wait complete. 2024-06-01T00:52:23.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:23 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/470127821' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:23.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:23 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/470127821' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:24 smithi160 ceph-mon[36639]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:24.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:24 smithi079 ceph-mon[29356]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:26.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:26 smithi160 ceph-mon[36639]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:26.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:26 smithi079 ceph-mon[29356]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:26.976 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:27.199 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:28 smithi079 ceph-mon[29356]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:28.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:28 smithi160 ceph-mon[36639]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2061981515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 msgr2=0x7f4d581008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2061981515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d581008a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009c20 tx=0x7f4d5002a970 comp rx=0 tx=0).stop 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2061981515 shutdown_connections 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2061981515 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d58100e70 0x7f4d58104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2061981515 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d581008a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2061981515 >> 172.21.15.79:0/2061981515 conn(0x7f4d580fbd70 msgr2=0x7f4d580fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2061981515 shutdown_connections 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2061981515 wait complete. 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.009+0000 7f4d5fb2c640 1 Processor -- start 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5fb2c640 1 -- start start 2024-06-01T00:52:29.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5fb2c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d58196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:29.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5fb2c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d58100e70 0x7f4d58196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:29.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5fb2c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d58196f10 con 0x7f4d581004c0 2024-06-01T00:52:29.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5fb2c640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5819ac30 con 0x7f4d58100e70 2024-06-01T00:52:29.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d8c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d58196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:29.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d8c8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d58196340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55376/0 (socket says 172.21.15.79:55376) 2024-06-01T00:52:29.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d8c8640 1 -- 172.21.15.79:0/2177198377 learned_addr learned my addr 172.21.15.79:0/2177198377 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d0c7640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d58100e70 0x7f4d58196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d8c8640 1 -- 172.21.15.79:0/2177198377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d58100e70 msgr2=0x7f4d58196880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d8c8640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d58100e70 0x7f4d58196880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d8c8640 1 -- 172.21.15.79:0/2177198377 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d50009900 con 0x7f4d581004c0 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d0c7640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d58100e70 0x7f4d58196880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.010+0000 7f4d5d8c8640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d58196340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4d5002ae90 tx=0x7f4d500308a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.011+0000 7f4d56ffd640 1 -- 172.21.15.79:0/2177198377 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d5002f590 con 0x7f4d581004c0 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.011+0000 7f4d56ffd640 1 -- 172.21.15.79:0/2177198377 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d50030ca0 con 0x7f4d581004c0 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.011+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5819aeb0 con 0x7f4d581004c0 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.011+0000 7f4d56ffd640 1 -- 172.21.15.79:0/2177198377 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d50039680 con 0x7f4d581004c0 2024-06-01T00:52:29.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.011+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5819b3a0 con 0x7f4d581004c0 2024-06-01T00:52:29.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.012+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d581019a0 con 0x7f4d581004c0 2024-06-01T00:52:29.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.012+0000 7f4d56ffd640 1 -- 172.21.15.79:0/2177198377 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f4d5000a040 con 0x7f4d581004c0 2024-06-01T00:52:29.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.013+0000 7f4d56ffd640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4d4006aef0 0x7f4d4006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:29.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.013+0000 7f4d56ffd640 1 -- 172.21.15.79:0/2177198377 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4d5002c040 con 0x7f4d581004c0 2024-06-01T00:52:29.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.013+0000 7f4d5d0c7640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4d4006aef0 0x7f4d4006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:29.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.013+0000 7f4d5d0c7640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4d4006aef0 0x7f4d4006d3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c006fd0 tx=0x7f4d4c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:29.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.017+0000 7f4d56ffd640 1 -- 172.21.15.79:0/2177198377 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f4d500765f0 con 0x7f4d581004c0 2024-06-01T00:52:29.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.223+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4d5804e210 con 0x7f4d581004c0 2024-06-01T00:52:29.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.224+0000 7f4d56ffd640 1 -- 172.21.15.79:0/2177198377 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4d50079c10 con 0x7f4d581004c0 2024-06-01T00:52:29.225 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:29.226 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4d4006aef0 msgr2=0x7f4d4006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4d4006aef0 0x7f4d4006d3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c006fd0 tx=0x7f4d4c008040 comp rx=0 tx=0).stop 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 msgr2=0x7f4d58196340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d58196340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4d5002ae90 tx=0x7f4d500308a0 comp rx=0 tx=0).stop 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 shutdown_connections 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f4d4006aef0 0x7f4d4006d3b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f4d58100e70 0x7f4d58196880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 --2- 172.21.15.79:0/2177198377 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4d581004c0 0x7f4d58196340 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:29.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 >> 172.21.15.79:0/2177198377 conn(0x7f4d580fbd70 msgr2=0x7f4d580fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:29.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 shutdown_connections 2024-06-01T00:52:29.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:29.226+0000 7f4d5fb2c640 1 -- 172.21.15.79:0/2177198377 wait complete. 2024-06-01T00:52:29.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:29 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2177198377' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:29.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:29 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2177198377' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:30.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:30 smithi160 ceph-mon[36639]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:30.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:30 smithi079 ceph-mon[29356]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:32 smithi160 ceph-mon[36639]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:32.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:32 smithi079 ceph-mon[29356]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:32.899 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:33.116 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:34.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 -- 172.21.15.79:0/523274658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ff3b0 msgr2=0x7f17d40ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:34.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 --2- 172.21.15.79:0/523274658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ff3b0 0x7f17d40ff790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f17c8009880 tx=0x7f17c802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 -- 172.21.15.79:0/523274658 shutdown_connections 2024-06-01T00:52:34.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 --2- 172.21.15.79:0/523274658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17d40ffd60 0x7f17d4103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 --2- 172.21.15.79:0/523274658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ff3b0 0x7f17d40ff790 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 -- 172.21.15.79:0/523274658 >> 172.21.15.79:0/523274658 conn(0x7f17d40fac80 msgr2=0x7f17d40fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:34.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 -- 172.21.15.79:0/523274658 shutdown_connections 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.646+0000 7f17dab3e640 1 -- 172.21.15.79:0/523274658 wait complete. 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.647+0000 7f17dab3e640 1 Processor -- start 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.647+0000 7f17dab3e640 1 -- start start 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.647+0000 7f17dab3e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17d40ff3b0 0x7f17d4100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.647+0000 7f17dab3e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ffd60 0x7f17d4101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.647+0000 7f17dab3e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d4102dd0 con 0x7f17d40ffd60 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.647+0000 7f17dab3e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d41018d0 con 0x7f17d40ff3b0 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ffd60 0x7f17d4101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ffd60 0x7f17d4101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34446/0 (socket says 172.21.15.79:34446) 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d3fff640 1 -- 172.21.15.79:0/3203436429 learned_addr learned my addr 172.21.15.79:0/3203436429 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:34.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d88da640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17d40ff3b0 0x7f17d4100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d3fff640 1 -- 172.21.15.79:0/3203436429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17d40ff3b0 msgr2=0x7f17d4100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d3fff640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17d40ff3b0 0x7f17d4100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d3fff640 1 -- 172.21.15.79:0/3203436429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8009560 con 0x7f17d40ffd60 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d3fff640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ffd60 0x7f17d4101390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c8f0 tx=0x7f17c000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d1ffb640 1 -- 172.21.15.79:0/3203436429 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17c0004410 con 0x7f17d40ffd60 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d4101b60 con 0x7f17d40ffd60 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d1ffb640 1 -- 172.21.15.79:0/3203436429 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17c0007c60 con 0x7f17d40ffd60 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17d1ffb640 1 -- 172.21.15.79:0/3203436429 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17c000f450 con 0x7f17d40ffd60 2024-06-01T00:52:34.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.648+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d41a26c0 con 0x7f17d40ffd60 2024-06-01T00:52:34.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.649+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d4100890 con 0x7f17d40ffd60 2024-06-01T00:52:34.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.650+0000 7f17d1ffb640 1 -- 172.21.15.79:0/3203436429 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f17c00077a0 con 0x7f17d40ffd60 2024-06-01T00:52:34.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.650+0000 7f17d1ffb640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f17c406aef0 0x7f17c406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:34.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.650+0000 7f17d1ffb640 1 -- 172.21.15.79:0/3203436429 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f17c0089bc0 con 0x7f17d40ffd60 2024-06-01T00:52:34.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.650+0000 7f17d88da640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f17c406aef0 0x7f17c406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:34.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.651+0000 7f17d88da640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f17c406aef0 0x7f17c406d3b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17c8000c00 tx=0x7f17c8005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:34.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.655+0000 7f17d1ffb640 1 -- 172.21.15.79:0/3203436429 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f17c00551a0 con 0x7f17d40ffd60 2024-06-01T00:52:34.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:34 smithi160 ceph-mon[36639]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:34.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:34 smithi079 ceph-mon[29356]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:34.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.862+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17d404e210 con 0x7f17d40ffd60 2024-06-01T00:52:34.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.862+0000 7f17d1ffb640 1 -- 172.21.15.79:0/3203436429 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f17c00587c0 con 0x7f17d40ffd60 2024-06-01T00:52:34.863 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:34.863 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:34.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f17c406aef0 msgr2=0x7f17c406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:34.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f17c406aef0 0x7f17c406d3b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f17c8000c00 tx=0x7f17c8005a20 comp rx=0 tx=0).stop 2024-06-01T00:52:34.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ffd60 msgr2=0x7f17d4101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:34.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ffd60 0x7f17d4101390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f17c000c8f0 tx=0x7f17c000cdb0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 shutdown_connections 2024-06-01T00:52:34.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f17c406aef0 0x7f17c406d3b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17d40ffd60 0x7f17d4101390 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 --2- 172.21.15.79:0/3203436429 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f17d40ff3b0 0x7f17d4100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:34.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 >> 172.21.15.79:0/3203436429 conn(0x7f17d40fac80 msgr2=0x7f17d40fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:34.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 shutdown_connections 2024-06-01T00:52:34.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:34.864+0000 7f17dab3e640 1 -- 172.21.15.79:0/3203436429 wait complete. 2024-06-01T00:52:35.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:35 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3203436429' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:35.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:35 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3203436429' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:36 smithi160 ceph-mon[36639]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:36.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:36 smithi079 ceph-mon[29356]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:38.451 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:38.666 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:38.788 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:38 smithi079 ceph-mon[29356]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:38 smithi160 ceph-mon[36639]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.267+0000 7fe53208b640 1 -- 172.21.15.79:0/2191779213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 msgr2=0x7fe52c104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 --2- 172.21.15.79:0/2191779213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c104e60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe520009880 tx=0x7fe52002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 -- 172.21.15.79:0/2191779213 shutdown_connections 2024-06-01T00:52:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 --2- 172.21.15.79:0/2191779213 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c104e60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 --2- 172.21.15.79:0/2191779213 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe52c100460 0x7fe52c100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 -- 172.21.15.79:0/2191779213 >> 172.21.15.79:0/2191779213 conn(0x7fe52c0fbd10 msgr2=0x7fe52c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 -- 172.21.15.79:0/2191779213 shutdown_connections 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 -- 172.21.15.79:0/2191779213 wait complete. 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.268+0000 7fe53208b640 1 Processor -- start 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe53208b640 1 -- start start 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe53208b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe52c100460 0x7fe52c196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe53208b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe53208b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c196ed0 con 0x7fe52c100e10 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe53208b640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c19abf0 con 0x7fe52c100460 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe52affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe52affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34460/0 (socket says 172.21.15.79:34460) 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe52affd640 1 -- 172.21.15.79:0/4001166072 learned_addr learned my addr 172.21.15.79:0/4001166072 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe52affd640 1 -- 172.21.15.79:0/4001166072 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe52c100460 msgr2=0x7fe52c196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:40.273 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe52affd640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe52c100460 0x7fe52c196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe52affd640 1 -- 172.21.15.79:0/4001166072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe520009560 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe52affd640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c196840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe52002a9c0 tx=0x7fe5200055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe528ff9640 1 -- 172.21.15.79:0/4001166072 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe520004aa0 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.269+0000 7fe528ff9640 1 -- 172.21.15.79:0/4001166072 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe520004c00 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.270+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c19ae70 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.270+0000 7fe528ff9640 1 -- 172.21.15.79:0/4001166072 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe520003a50 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.270+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c19b360 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.270+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c101940 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.271+0000 7fe528ff9640 1 -- 172.21.15.79:0/4001166072 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe520031c90 con 0x7fe52c100e10 2024-06-01T00:52:40.274 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.271+0000 7fe528ff9640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe51406aef0 0x7fe51406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:40.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.271+0000 7fe528ff9640 1 -- 172.21.15.79:0/4001166072 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe5200aae00 con 0x7fe52c100e10 2024-06-01T00:52:40.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.271+0000 7fe52b7fe640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe51406aef0 0x7fe51406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:40.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.272+0000 7fe52b7fe640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe51406aef0 0x7fe51406d3b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe52400ab90 tx=0x7fe524009250 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:40.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.276+0000 7fe528ff9640 1 -- 172.21.15.79:0/4001166072 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe5200763e0 con 0x7fe52c100e10 2024-06-01T00:52:40.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.487+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe52c04e210 con 0x7fe52c100e10 2024-06-01T00:52:40.488 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:40.488 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:40.488 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.487+0000 7fe528ff9640 1 -- 172.21.15.79:0/4001166072 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe520079a00 con 0x7fe52c100e10 2024-06-01T00:52:40.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe51406aef0 msgr2=0x7fe51406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe51406aef0 0x7fe51406d3b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe52400ab90 tx=0x7fe524009250 comp rx=0 tx=0).stop 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 msgr2=0x7fe52c196840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c196840 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe52002a9c0 tx=0x7fe5200055a0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 shutdown_connections 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe51406aef0 0x7fe51406d3b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe52c100e10 0x7fe52c196840 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 --2- 172.21.15.79:0/4001166072 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe52c100460 0x7fe52c196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 >> 172.21.15.79:0/4001166072 conn(0x7fe52c0fbd10 msgr2=0x7fe52c0fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 shutdown_connections 2024-06-01T00:52:40.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:40.489+0000 7fe53208b640 1 -- 172.21.15.79:0/4001166072 wait complete. 2024-06-01T00:52:40.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:40 smithi160 ceph-mon[36639]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:40.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:40 smithi079 ceph-mon[29356]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:41.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:41 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4001166072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:41.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:41 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4001166072' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:42.657 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:42 smithi079 ceph-mon[29356]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:42.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:42 smithi160 ceph-mon[36639]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:44.206 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:44.423 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:44.543 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:44 smithi079 ceph-mon[29356]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:44 smithi160 ceph-mon[36639]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:46.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 -- 172.21.15.79:0/379283341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 msgr2=0x7f2754100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 --2- 172.21.15.79:0/379283341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754100850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f274c009880 tx=0x7f274c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 -- 172.21.15.79:0/379283341 shutdown_connections 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 --2- 172.21.15.79:0/379283341 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2754100e20 0x7f2754104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 --2- 172.21.15.79:0/379283341 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754100850 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 -- 172.21.15.79:0/379283341 >> 172.21.15.79:0/379283341 conn(0x7f27540fbd20 msgr2=0x7f27540fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 -- 172.21.15.79:0/379283341 shutdown_connections 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.020+0000 7f275b117640 1 -- 172.21.15.79:0/379283341 wait complete. 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f275b117640 1 Processor -- start 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f275b117640 1 -- start start 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f275b117640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:46.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f275b117640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2754100e20 0x7f2754196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f275b117640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754196e20 con 0x7f2754100470 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f275b117640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275419aaf0 con 0x7f2754100e20 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f2758eb3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f2758eb3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754196250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60804/0 (socket says 172.21.15.79:60804) 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f2758eb3640 1 -- 172.21.15.79:0/695587853 learned_addr learned my addr 172.21.15.79:0/695587853 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f2758eb3640 1 -- 172.21.15.79:0/695587853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2754100e20 msgr2=0x7f2754196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f2758eb3640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2754100e20 0x7f2754196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.021+0000 7f2758eb3640 1 -- 172.21.15.79:0/695587853 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c009560 con 0x7f2754100470 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.022+0000 7f2758eb3640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754196250 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f274c02a9c0 tx=0x7f274c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.022+0000 7f2751ffb640 1 -- 172.21.15.79:0/695587853 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f274c004aa0 con 0x7f2754100470 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.022+0000 7f2751ffb640 1 -- 172.21.15.79:0/695587853 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f274c004c00 con 0x7f2754100470 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.022+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275419ad70 con 0x7f2754100470 2024-06-01T00:52:46.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.022+0000 7f2751ffb640 1 -- 172.21.15.79:0/695587853 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f274c003980 con 0x7f2754100470 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.022+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f275419b260 con 0x7f2754100470 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.023+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2754101950 con 0x7f2754100470 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.024+0000 7f2751ffb640 1 -- 172.21.15.79:0/695587853 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f274c003b20 con 0x7f2754100470 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.024+0000 7f2751ffb640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f274006aef0 0x7f274006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.024+0000 7f2751ffb640 1 -- 172.21.15.79:0/695587853 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f274c0aaba0 con 0x7f2754100470 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.024+0000 7f2753fff640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f274006aef0 0x7f274006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.024+0000 7f2753fff640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f274006aef0 0x7f274006d3b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2754197760 tx=0x7f2744009710 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:46.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.028+0000 7f2751ffb640 1 -- 172.21.15.79:0/695587853 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f274c076200 con 0x7f2754100470 2024-06-01T00:52:46.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.237+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f275404e210 con 0x7f2754100470 2024-06-01T00:52:46.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.238+0000 7f2751ffb640 1 -- 172.21.15.79:0/695587853 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f274c079820 con 0x7f2754100470 2024-06-01T00:52:46.239 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:46.239 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f274006aef0 msgr2=0x7f274006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f274006aef0 0x7f274006d3b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2754197760 tx=0x7f2744009710 comp rx=0 tx=0).stop 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 msgr2=0x7f2754196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754196250 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f274c02a9c0 tx=0x7f274c0055a0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 shutdown_connections 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f274006aef0 0x7f274006d3b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2754100e20 0x7f2754196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 --2- 172.21.15.79:0/695587853 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2754100470 0x7f2754196250 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 >> 172.21.15.79:0/695587853 conn(0x7f27540fbd20 msgr2=0x7f27540fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 shutdown_connections 2024-06-01T00:52:46.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:46.240+0000 7f275b117640 1 -- 172.21.15.79:0/695587853 wait complete. 2024-06-01T00:52:46.547 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:46 smithi079 ceph-mon[29356]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:46.547 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:46 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/695587853' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:46.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:46 smithi160 ceph-mon[36639]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:46.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:46 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/695587853' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:48.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:48 smithi160 ceph-mon[36639]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:48.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:48 smithi079 ceph-mon[29356]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:49.861 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:50.075 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:50 smithi160 ceph-mon[36639]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:50 smithi079 ceph-mon[29356]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:52:51.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 -- 172.21.15.79:0/915128995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ffd40 msgr2=0x7f6e9c103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:51.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 --2- 172.21.15.79:0/915128995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ffd40 0x7f6e9c103d90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009c20 tx=0x7f6e9002a970 comp rx=0 tx=0).stop 2024-06-01T00:52:51.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 -- 172.21.15.79:0/915128995 shutdown_connections 2024-06-01T00:52:51.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 --2- 172.21.15.79:0/915128995 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ffd40 0x7f6e9c103d90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:51.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 --2- 172.21.15.79:0/915128995 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c0ff390 0x7f6e9c0ff770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:51.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 -- 172.21.15.79:0/915128995 >> 172.21.15.79:0/915128995 conn(0x7f6e9c0fac40 msgr2=0x7f6e9c0fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:51.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 -- 172.21.15.79:0/915128995 shutdown_connections 2024-06-01T00:52:51.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.642+0000 7f6ea0871640 1 -- 172.21.15.79:0/915128995 wait complete. 2024-06-01T00:52:51.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6ea0871640 1 Processor -- start 2024-06-01T00:52:51.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6ea0871640 1 -- start start 2024-06-01T00:52:51.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6ea0871640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ff390 0x7f6e9c194eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6ea0871640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c0ffd40 0x7f6e9c1953f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6ea0871640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c1959c0 con 0x7f6e9c0ff390 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6ea0871640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9c195b30 con 0x7f6e9c0ffd40 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6e9a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ff390 0x7f6e9c194eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6e9a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ff390 0x7f6e9c194eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60826/0 (socket says 172.21.15.79:60826) 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6e9a59c640 1 -- 172.21.15.79:0/1156457408 learned_addr learned my addr 172.21.15.79:0/1156457408 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.643+0000 7f6e99d9b640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c0ffd40 0x7f6e9c1953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6e9a59c640 1 -- 172.21.15.79:0/1156457408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c0ffd40 msgr2=0x7f6e9c1953f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6e9a59c640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c0ffd40 0x7f6e9c1953f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:51.647 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6e9a59c640 1 -- 172.21.15.79:0/1156457408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e90009900 con 0x7f6e9c0ff390 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6e9a59c640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ff390 0x7f6e9c194eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400d8d0 tx=0x7f6e9400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6e8b7fe640 1 -- 172.21.15.79:0/1156457408 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9400bea0 con 0x7f6e9c0ff390 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9c199f00 con 0x7f6e9c0ff390 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6e8b7fe640 1 -- 172.21.15.79:0/1156457408 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e94009c00 con 0x7f6e9c0ff390 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6e8b7fe640 1 -- 172.21.15.79:0/1156457408 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e9400f710 con 0x7f6e9c0ff390 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.644+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9c19a420 con 0x7f6e9c0ff390 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.645+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9c100870 con 0x7f6e9c0ff390 2024-06-01T00:52:51.648 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.646+0000 7f6e8b7fe640 1 -- 172.21.15.79:0/1156457408 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6e9400f990 con 0x7f6e9c0ff390 2024-06-01T00:52:51.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.646+0000 7f6e8b7fe640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6e8406aef0 0x7f6e8406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:51.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.646+0000 7f6e99d9b640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6e8406aef0 0x7f6e8406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:51.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.646+0000 7f6e8b7fe640 1 -- 172.21.15.79:0/1156457408 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6e940898c0 con 0x7f6e9c0ff390 2024-06-01T00:52:51.649 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.647+0000 7f6e99d9b640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6e8406aef0 0x7f6e8406d3b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6e90000f80 tx=0x7f6e9003d040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:51.651 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.651+0000 7f6e8b7fe640 1 -- 172.21.15.79:0/1156457408 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6e94054f50 con 0x7f6e9c0ff390 2024-06-01T00:52:51.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.857+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6e9c04e210 con 0x7f6e9c0ff390 2024-06-01T00:52:51.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.857+0000 7f6e8b7fe640 1 -- 172.21.15.79:0/1156457408 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6e94058570 con 0x7f6e9c0ff390 2024-06-01T00:52:51.858 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:51.858 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:51.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6e8406aef0 msgr2=0x7f6e8406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:51.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6e8406aef0 0x7f6e8406d3b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6e90000f80 tx=0x7f6e9003d040 comp rx=0 tx=0).stop 2024-06-01T00:52:51.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ff390 msgr2=0x7f6e9c194eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:51.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ff390 0x7f6e9c194eb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400d8d0 tx=0x7f6e9400dd90 comp rx=0 tx=0).stop 2024-06-01T00:52:51.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 shutdown_connections 2024-06-01T00:52:51.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6e8406aef0 0x7f6e8406d3b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:51.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6e9c0ffd40 0x7f6e9c1953f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:51.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 --2- 172.21.15.79:0/1156457408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e9c0ff390 0x7f6e9c194eb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:51.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 >> 172.21.15.79:0/1156457408 conn(0x7f6e9c0fac40 msgr2=0x7f6e9c102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:51.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 shutdown_connections 2024-06-01T00:52:51.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:51.859+0000 7f6ea0871640 1 -- 172.21.15.79:0/1156457408 wait complete. 2024-06-01T00:52:52.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:52 smithi079 ceph-mon[29356]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:52.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:52 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1156457408' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:52 smithi160 ceph-mon[36639]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:52 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1156457408' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:54.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:54 smithi160 ceph-mon[36639]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:54.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:54 smithi079 ceph-mon[29356]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:55.434 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:52:55.651 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:52:56.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:56 smithi160 ceph-mon[36639]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:56.811 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:56 smithi079 ceph-mon[29356]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 -- 172.21.15.79:0/1556555650 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45500fe8c0 msgr2=0x7f45500fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 --2- 172.21.15.79:0/1556555650 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45500fe8c0 0x7f45500fed40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f454c00a990 tx=0x7f454c02b7d0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 -- 172.21.15.79:0/1556555650 shutdown_connections 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 --2- 172.21.15.79:0/1556555650 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45500fe8c0 0x7f45500fed40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 --2- 172.21.15.79:0/1556555650 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f45501048c0 0x7f4550104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 -- 172.21.15.79:0/1556555650 >> 172.21.15.79:0/1556555650 conn(0x7f4550075570 msgr2=0x7f45500fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 -- 172.21.15.79:0/1556555650 shutdown_connections 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 -- 172.21.15.79:0/1556555650 wait complete. 2024-06-01T00:52:57.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 Processor -- start 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.137+0000 7f45576a7640 1 -- start start 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45576a7640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f45500fe8c0 0x7f45501962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45576a7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45501048c0 0x7f45501967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45576a7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550196e70 con 0x7f45501048c0 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45576a7640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455019ab20 con 0x7f45500fe8c0 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4554c42640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45501048c0 0x7f45501967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4554c42640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45501048c0 0x7f45501967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:41824/0 (socket says 172.21.15.79:41824) 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4554c42640 1 -- 172.21.15.79:0/4030174300 learned_addr learned my addr 172.21.15.79:0/4030174300 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4555443640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f45500fe8c0 0x7f45501962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4554c42640 1 -- 172.21.15.79:0/4030174300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f45500fe8c0 msgr2=0x7f45501962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4554c42640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f45500fe8c0 0x7f45501962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4554c42640 1 -- 172.21.15.79:0/4030174300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c00a670 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f4554c42640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45501048c0 0x7f45501967e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f454c02bcf0 tx=0x7f454c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45467fc640 1 -- 172.21.15.79:0/4030174300 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f454c004420 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455019ada0 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45467fc640 1 -- 172.21.15.79:0/4030174300 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f454c004ae0 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45467fc640 1 -- 172.21.15.79:0/4030174300 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f454c005340 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.138+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455019b2f0 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.139+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4550100000 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.140+0000 7f45467fc640 1 -- 172.21.15.79:0/4030174300 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f454c0035f0 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.140+0000 7f45467fc640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f453c06aef0 0x7f453c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.140+0000 7f45467fc640 1 -- 172.21.15.79:0/4030174300 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f454c0ab120 con 0x7f45501048c0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.140+0000 7f4555443640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f453c06aef0 0x7f453c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.141+0000 7f4555443640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f453c06aef0 0x7f453c06d3b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4540009990 tx=0x7f4540009110 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:52:57.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.145+0000 7f45467fc640 1 -- 172.21.15.79:0/4030174300 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f454c076820 con 0x7f45501048c0 2024-06-01T00:52:57.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.353+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f455004e210 con 0x7f45501048c0 2024-06-01T00:52:57.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.353+0000 7f45467fc640 1 -- 172.21.15.79:0/4030174300 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f454c079e40 con 0x7f45501048c0 2024-06-01T00:52:57.355 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:52:57.355 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f453c06aef0 msgr2=0x7f453c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f453c06aef0 0x7f453c06d3b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4540009990 tx=0x7f4540009110 comp rx=0 tx=0).stop 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45501048c0 msgr2=0x7f45501967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45501048c0 0x7f45501967e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f454c02bcf0 tx=0x7f454c0048f0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 shutdown_connections 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f453c06aef0 0x7f453c06d3b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f45501048c0 0x7f45501967e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.357 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 --2- 172.21.15.79:0/4030174300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f45500fe8c0 0x7f45501962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:52:57.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 >> 172.21.15.79:0/4030174300 conn(0x7f4550075570 msgr2=0x7f45500fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:52:57.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 shutdown_connections 2024-06-01T00:52:57.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:52:57.355+0000 7f45576a7640 1 -- 172.21.15.79:0/4030174300 wait complete. 2024-06-01T00:52:57.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:57 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4030174300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:57.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:57 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4030174300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:52:58.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:52:58 smithi160 ceph-mon[36639]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:52:58.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:52:58 smithi079 ceph-mon[29356]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:00.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:00 smithi160 ceph-mon[36639]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:00.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:00 smithi079 ceph-mon[29356]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:01.059 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:01.276 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:02.657 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:02 smithi079 ceph-mon[29356]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:02.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:02 smithi160 ceph-mon[36639]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 -- 172.21.15.79:0/2769019251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101d30 msgr2=0x7f204810a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 --2- 172.21.15.79:0/2769019251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101d30 0x7f204810a220 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2044009880 tx=0x7f204402a530 comp rx=0 tx=0).stop 2024-06-01T00:53:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 -- 172.21.15.79:0/2769019251 shutdown_connections 2024-06-01T00:53:02.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 --2- 172.21.15.79:0/2769019251 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101d30 0x7f204810a220 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 --2- 172.21.15.79:0/2769019251 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2048101410 0x7f20481017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 -- 172.21.15.79:0/2769019251 >> 172.21.15.79:0/2769019251 conn(0x7f20480fac80 msgr2=0x7f20480fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 -- 172.21.15.79:0/2769019251 shutdown_connections 2024-06-01T00:53:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.858+0000 7f204fc52640 1 -- 172.21.15.79:0/2769019251 wait complete. 2024-06-01T00:53:02.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204fc52640 1 Processor -- start 2024-06-01T00:53:02.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204fc52640 1 -- start start 2024-06-01T00:53:02.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204fc52640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101410 0x7f2048196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:02.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204fc52640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2048101d30 0x7f2048196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:02.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204fc52640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2048196f00 con 0x7f2048101410 2024-06-01T00:53:02.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204fc52640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204819ac20 con 0x7f2048101d30 2024-06-01T00:53:02.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d9ee640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101410 0x7f2048196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:02.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d9ee640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101410 0x7f2048196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49322/0 (socket says 172.21.15.79:49322) 2024-06-01T00:53:02.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d9ee640 1 -- 172.21.15.79:0/3779681531 learned_addr learned my addr 172.21.15.79:0/3779681531 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:02.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d1ed640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2048101d30 0x7f2048196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:02.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d9ee640 1 -- 172.21.15.79:0/3779681531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2048101d30 msgr2=0x7f2048196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:02.867 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d9ee640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2048101d30 0x7f2048196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:02.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d9ee640 1 -- 172.21.15.79:0/3779681531 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2044009560 con 0x7f2048101410 2024-06-01T00:53:02.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.859+0000 7f204d1ed640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2048101d30 0x7f2048196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:53:02.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.860+0000 7f204d9ee640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101410 0x7f2048196330 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f203800b420 tx=0x7f203800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:02.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.860+0000 7f203effd640 1 -- 172.21.15.79:0/3779681531 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2038011800 con 0x7f2048101410 2024-06-01T00:53:02.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.860+0000 7f203effd640 1 -- 172.21.15.79:0/3779681531 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2038011e20 con 0x7f2048101410 2024-06-01T00:53:02.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.860+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204819af00 con 0x7f2048101410 2024-06-01T00:53:02.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.860+0000 7f203effd640 1 -- 172.21.15.79:0/3779681531 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f203800f2e0 con 0x7f2048101410 2024-06-01T00:53:02.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.860+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204819b450 con 0x7f2048101410 2024-06-01T00:53:02.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.861+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2048069060 con 0x7f2048101410 2024-06-01T00:53:02.869 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.861+0000 7f203effd640 1 -- 172.21.15.79:0/3779681531 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2038010380 con 0x7f2048101410 2024-06-01T00:53:02.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.862+0000 7f203effd640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f203406aef0 0x7f203406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:02.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.862+0000 7f204d1ed640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f203406aef0 0x7f203406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:02.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.862+0000 7f203effd640 1 -- 172.21.15.79:0/3779681531 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f20380894d0 con 0x7f2048101410 2024-06-01T00:53:02.870 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.862+0000 7f204d1ed640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f203406aef0 0x7f203406d3b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f2044005e60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:02.871 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:02.866+0000 7f203effd640 1 -- 172.21.15.79:0/3779681531 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2038054b30 con 0x7f2048101410 2024-06-01T00:53:03.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.102+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f204804e210 con 0x7f2048101410 2024-06-01T00:53:03.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.103+0000 7f203effd640 1 -- 172.21.15.79:0/3779681531 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2038058150 con 0x7f2048101410 2024-06-01T00:53:03.104 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:03.104 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f203406aef0 msgr2=0x7f203406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f203406aef0 0x7f203406d3b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f2044005e60 comp rx=0 tx=0).stop 2024-06-01T00:53:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101410 msgr2=0x7f2048196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101410 0x7f2048196330 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f203800b420 tx=0x7f203800b8e0 comp rx=0 tx=0).stop 2024-06-01T00:53:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 shutdown_connections 2024-06-01T00:53:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f203406aef0 0x7f203406d3b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:03.106 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f2048101d30 0x7f2048196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 --2- 172.21.15.79:0/3779681531 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2048101410 0x7f2048196330 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 >> 172.21.15.79:0/3779681531 conn(0x7f20480fac80 msgr2=0x7f2048102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 shutdown_connections 2024-06-01T00:53:03.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:03.105+0000 7f204fc52640 1 -- 172.21.15.79:0/3779681531 wait complete. 2024-06-01T00:53:03.684 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:03 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3779681531' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:03.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:03 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3779681531' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:04.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:04 smithi160 ceph-mon[36639]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:04.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:04 smithi079 ceph-mon[29356]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:05.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:05.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:06.706 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:06 smithi160 ceph-mon[36639]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:53:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:53:06.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:53:06.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:53:06.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:06 smithi079 ceph-mon[29356]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:53:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:53:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:53:06.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:53:06.910 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:08.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 -- 172.21.15.79:0/4025719965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 msgr2=0x7f9278104ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:08.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 --2- 172.21.15.79:0/4025719965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f9278104ca0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f926800a990 tx=0x7f926802b7d0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 -- 172.21.15.79:0/4025719965 shutdown_connections 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 --2- 172.21.15.79:0/4025719965 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f92780fea40 0x7f92780feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 --2- 172.21.15.79:0/4025719965 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f9278104ca0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 -- 172.21.15.79:0/4025719965 >> 172.21.15.79:0/4025719965 conn(0x7f9278074460 msgr2=0x7f9278074870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 -- 172.21.15.79:0/4025719965 shutdown_connections 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 -- 172.21.15.79:0/4025719965 wait complete. 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 Processor -- start 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.530+0000 7f927dc4a640 1 -- start start 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f927dc4a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f92780fea40 0x7f9278198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f927dc4a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f92781988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f927dc4a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9278198f50 con 0x7f92781048c0 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f927dc4a640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927819cc70 con 0x7f92780fea40 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f926effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f92781988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:08.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f926effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f92781988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49344/0 (socket says 172.21.15.79:49344) 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f926effd640 1 -- 172.21.15.79:0/2477206189 learned_addr learned my addr 172.21.15.79:0/2477206189 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f92777fe640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f92780fea40 0x7f9278198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f926effd640 1 -- 172.21.15.79:0/2477206189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f92780fea40 msgr2=0x7f9278198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f926effd640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f92780fea40 0x7f9278198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f926effd640 1 -- 172.21.15.79:0/2477206189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926800a670 con 0x7f92781048c0 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f926effd640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f92781988c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f926000c810 tx=0x7f926000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f92757fa640 1 -- 172.21.15.79:0/2477206189 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92600041f0 con 0x7f92781048c0 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927819cf50 con 0x7f92781048c0 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.531+0000 7f92757fa640 1 -- 172.21.15.79:0/2477206189 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9260007e70 con 0x7f92781048c0 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.532+0000 7f92757fa640 1 -- 172.21.15.79:0/2477206189 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92600049d0 con 0x7f92781048c0 2024-06-01T00:53:08.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.532+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927819d4a0 con 0x7f92781048c0 2024-06-01T00:53:08.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.532+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9278100020 con 0x7f92781048c0 2024-06-01T00:53:08.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.533+0000 7f92757fa640 1 -- 172.21.15.79:0/2477206189 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f92600079b0 con 0x7f92781048c0 2024-06-01T00:53:08.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.534+0000 7f92757fa640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f926406aef0 0x7f926406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:08.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.534+0000 7f92777fe640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f926406aef0 0x7f926406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:08.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.534+0000 7f92757fa640 1 -- 172.21.15.79:0/2477206189 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9260089a50 con 0x7f92781048c0 2024-06-01T00:53:08.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.534+0000 7f92777fe640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f926406aef0 0x7f926406d3b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9268036040 tx=0x7f9268005fd0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:08.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.538+0000 7f92757fa640 1 -- 172.21.15.79:0/2477206189 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9260055030 con 0x7f92781048c0 2024-06-01T00:53:08.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:08 smithi079 ceph-mon[29356]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:08.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.749+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9278106e30 con 0x7f92781048c0 2024-06-01T00:53:08.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.749+0000 7f92757fa640 1 -- 172.21.15.79:0/2477206189 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f926001d0d0 con 0x7f92781048c0 2024-06-01T00:53:08.750 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:08.750 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:08.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f926406aef0 msgr2=0x7f926406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f926406aef0 0x7f926406d3b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9268036040 tx=0x7f9268005fd0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 msgr2=0x7f92781988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f92781988c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f926000c810 tx=0x7f926000ccd0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 shutdown_connections 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f926406aef0 0x7f926406d3b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f92781048c0 0x7f92781988c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 --2- 172.21.15.79:0/2477206189 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f92780fea40 0x7f9278198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.751+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 >> 172.21.15.79:0/2477206189 conn(0x7f9278074460 msgr2=0x7f92780fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.752+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 shutdown_connections 2024-06-01T00:53:08.753 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:08.752+0000 7f927dc4a640 1 -- 172.21.15.79:0/2477206189 wait complete. 2024-06-01T00:53:08.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:08 smithi160 ceph-mon[36639]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:09.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:09 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2477206189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:09.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:09 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2477206189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:10.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:10 smithi160 ceph-mon[36639]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:10.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:10 smithi079 ceph-mon[29356]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:12.454 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:12.669 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:12.791 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:12 smithi079 ceph-mon[29356]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:12 smithi160 ceph-mon[36639]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:14.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.409+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/2843651316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 msgr2=0x7f3fd00ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:14.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.409+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/2843651316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd00ff5a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0009880 tx=0x7f3fc002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/2843651316 shutdown_connections 2024-06-01T00:53:14.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/2843651316 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fd00ffb70 0x7f3fd0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/2843651316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd00ff5a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/2843651316 >> 172.21.15.79:0/2843651316 conn(0x7f3fd00faab0 msgr2=0x7f3fd00fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/2843651316 shutdown_connections 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/2843651316 wait complete. 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 Processor -- start 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 -- start start 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd0196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fd00ffb70 0x7f3fd0197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd0197980 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.410+0000 7f3fd4b9d640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd0197af0 con 0x7f3fd00ffb70 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd0196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd0196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36374/0 (socket says 172.21.15.79:36374) 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcf7fe640 1 -- 172.21.15.79:0/3322743447 learned_addr learned my addr 172.21.15.79:0/3322743447 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:14.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fc7fff640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fd00ffb70 0x7f3fd0197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcf7fe640 1 -- 172.21.15.79:0/3322743447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fd00ffb70 msgr2=0x7f3fd0197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcf7fe640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fd00ffb70 0x7f3fd0197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcf7fe640 1 -- 172.21.15.79:0/3322743447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc0009560 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcf7fe640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd0196f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3fc002a9c0 tx=0x7f3fc00055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcd7fa640 1 -- 172.21.15.79:0/3322743447 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fc00048d0 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcd7fa640 1 -- 172.21.15.79:0/3322743447 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fc0031640 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd0197d70 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fcd7fa640 1 -- 172.21.15.79:0/3322743447 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fc00038d0 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.411+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd01a0250 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.412+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd01006a0 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.413+0000 7f3fcd7fa640 1 -- 172.21.15.79:0/3322743447 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f3fc0004a30 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.413+0000 7f3fcd7fa640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3fbc06aef0 0x7f3fbc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:14.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.413+0000 7f3fcd7fa640 1 -- 172.21.15.79:0/3322743447 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3fc00aa920 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.413+0000 7f3fc7fff640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3fbc06aef0 0x7f3fbc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:14.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.414+0000 7f3fc7fff640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3fbc06aef0 0x7f3fbc06d3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0068c60 tx=0x7f3fb8005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:14.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.417+0000 7f3fcd7fa640 1 -- 172.21.15.79:0/3322743447 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3fc0076030 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.624+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3fd0197d70 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.625 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.624+0000 7f3fcd7fa640 1 -- 172.21.15.79:0/3322743447 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3fd0197d70 con 0x7f3fd00ff1c0 2024-06-01T00:53:14.625 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:14.625 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3fbc06aef0 msgr2=0x7f3fbc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3fbc06aef0 0x7f3fbc06d3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0068c60 tx=0x7f3fb8005f90 comp rx=0 tx=0).stop 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 msgr2=0x7f3fd0196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd0196f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3fc002a9c0 tx=0x7f3fc00055a0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 shutdown_connections 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3fbc06aef0 0x7f3fbc06d3b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3fd00ffb70 0x7f3fd0197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 --2- 172.21.15.79:0/3322743447 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3fd00ff1c0 0x7f3fd0196f00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:14.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 >> 172.21.15.79:0/3322743447 conn(0x7f3fd00faab0 msgr2=0x7f3fd00fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:14.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 shutdown_connections 2024-06-01T00:53:14.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:14.626+0000 7f3fd4b9d640 1 -- 172.21.15.79:0/3322743447 wait complete. 2024-06-01T00:53:14.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:14 smithi160 ceph-mon[36639]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:14.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:14 smithi079 ceph-mon[29356]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:15.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:15 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3322743447' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:15.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:15 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3322743447' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:16.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:16 smithi160 ceph-mon[36639]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:16.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:16 smithi079 ceph-mon[29356]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:18.367 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:18.585 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:18.706 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:18 smithi079 ceph-mon[29356]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:18 smithi160 ceph-mon[36639]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:20.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 -- 172.21.15.79:0/1604057464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ffd60 msgr2=0x7f32ac103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:20.241 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 --2- 172.21.15.79:0/1604057464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ffd60 0x7f32ac103db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009880 tx=0x7f32a002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 -- 172.21.15.79:0/1604057464 shutdown_connections 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 --2- 172.21.15.79:0/1604057464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ffd60 0x7f32ac103db0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 --2- 172.21.15.79:0/1604057464 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32ac0ff3b0 0x7f32ac0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 -- 172.21.15.79:0/1604057464 >> 172.21.15.79:0/1604057464 conn(0x7f32ac0fac80 msgr2=0x7f32ac0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 -- 172.21.15.79:0/1604057464 shutdown_connections 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 -- 172.21.15.79:0/1604057464 wait complete. 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 Processor -- start 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.239+0000 7f32b2d99640 1 -- start start 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b2d99640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ff3b0 0x7f32ac195110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b2d99640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32ac0ffd60 0x7f32ac195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b2d99640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac195c50 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.242 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b2d99640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac195dc0 con 0x7f32ac0ffd60 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b0b35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ff3b0 0x7f32ac195110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b0b35640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ff3b0 0x7f32ac195110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36396/0 (socket says 172.21.15.79:36396) 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b0b35640 1 -- 172.21.15.79:0/845802046 learned_addr learned my addr 172.21.15.79:0/845802046 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32abfff640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32ac0ffd60 0x7f32ac195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b0b35640 1 -- 172.21.15.79:0/845802046 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32ac0ffd60 msgr2=0x7f32ac195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b0b35640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32ac0ffd60 0x7f32ac195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b0b35640 1 -- 172.21.15.79:0/845802046 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0009560 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b0b35640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ff3b0 0x7f32ac195110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f329c007b50 tx=0x7f329c00e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32a9ffb640 1 -- 172.21.15.79:0/845802046 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329c011070 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac199b50 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32a9ffb640 1 -- 172.21.15.79:0/845802046 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f329c00c2e0 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.240+0000 7f32a9ffb640 1 -- 172.21.15.79:0/845802046 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329c00ae30 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.243 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.241+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac19a070 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.241+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac100890 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.242+0000 7f32a9ffb640 1 -- 172.21.15.79:0/845802046 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f329c00eab0 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.242+0000 7f32a9ffb640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f329406ae90 0x7f329406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:20.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.242+0000 7f32a9ffb640 1 -- 172.21.15.79:0/845802046 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f329c08b4d0 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.242+0000 7f32abfff640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f329406ae90 0x7f329406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:20.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.243+0000 7f32abfff640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f329406ae90 0x7f329406d350 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009230 tx=0x7f32a0005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:20.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.246+0000 7f32a9ffb640 1 -- 172.21.15.79:0/845802046 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f329c056ab0 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.456+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f32ac04e210 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.456+0000 7f32a9ffb640 1 -- 172.21.15.79:0/845802046 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f329c05a0d0 con 0x7f32ac0ff3b0 2024-06-01T00:53:20.457 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:20.458 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:20.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f329406ae90 msgr2=0x7f329406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:20.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f329406ae90 0x7f329406d350 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32a0009230 tx=0x7f32a0005a20 comp rx=0 tx=0).stop 2024-06-01T00:53:20.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ff3b0 msgr2=0x7f32ac195110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ff3b0 0x7f32ac195110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f329c007b50 tx=0x7f329c00e420 comp rx=0 tx=0).stop 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 shutdown_connections 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f329406ae90 0x7f329406d350 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f32ac0ffd60 0x7f32ac195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 --2- 172.21.15.79:0/845802046 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32ac0ff3b0 0x7f32ac195110 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.458+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 >> 172.21.15.79:0/845802046 conn(0x7f32ac0fac80 msgr2=0x7f32ac102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.459+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 shutdown_connections 2024-06-01T00:53:20.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:20.459+0000 7f32b2d99640 1 -- 172.21.15.79:0/845802046 wait complete. 2024-06-01T00:53:20.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:20 smithi160 ceph-mon[36639]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:20.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:20.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:20 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/845802046' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:20.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:20 smithi079 ceph-mon[29356]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:20.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:20.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:20 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/845802046' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:21.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:21 smithi079 ceph-mon[29356]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:22.001 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:21 smithi160 ceph-mon[36639]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:24.131 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:24.347 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:24.469 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:24 smithi079 ceph-mon[29356]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:24 smithi160 ceph-mon[36639]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:25.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 -- 172.21.15.79:0/3605365224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 msgr2=0x7fe6d8104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:25.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3605365224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8104eb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc009880 tx=0x7fe6cc02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:53:25.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 -- 172.21.15.79:0/3605365224 shutdown_connections 2024-06-01T00:53:25.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3605365224 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8104eb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:25.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3605365224 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6d81004b0 0x7fe6d8100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:25.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 -- 172.21.15.79:0/3605365224 >> 172.21.15.79:0/3605365224 conn(0x7fe6d80fbda0 msgr2=0x7fe6d80fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:25.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 -- 172.21.15.79:0/3605365224 shutdown_connections 2024-06-01T00:53:25.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.919+0000 7fe6dd975640 1 -- 172.21.15.79:0/3605365224 wait complete. 2024-06-01T00:53:25.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6dd975640 1 Processor -- start 2024-06-01T00:53:25.926 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6dd975640 1 -- start start 2024-06-01T00:53:25.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6dd975640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6d81004b0 0x7fe6d80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:25.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6dd975640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:25.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6dd975640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8073230 con 0x7fe6d8100e60 2024-06-01T00:53:25.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6dd975640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d80733a0 con 0x7fe6d81004b0 2024-06-01T00:53:25.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d67fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:25.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d67fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8072cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53470/0 (socket says 172.21.15.79:53470) 2024-06-01T00:53:25.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d67fc640 1 -- 172.21.15.79:0/3626813959 learned_addr learned my addr 172.21.15.79:0/3626813959 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:25.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d6ffd640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6d81004b0 0x7fe6d80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:25.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d67fc640 1 -- 172.21.15.79:0/3626813959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6d81004b0 msgr2=0x7fe6d80727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:25.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d67fc640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6d81004b0 0x7fe6d80727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:25.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d67fc640 1 -- 172.21.15.79:0/3626813959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc009560 con 0x7fe6d8100e60 2024-06-01T00:53:25.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.920+0000 7fe6d6ffd640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6d81004b0 0x7fe6d80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:53:25.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.921+0000 7fe6d67fc640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8072cf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc02a9c0 tx=0x7fe6cc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:25.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.921+0000 7fe6dc973640 1 -- 172.21.15.79:0/3626813959 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6cc0048d0 con 0x7fe6d8100e60 2024-06-01T00:53:25.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.921+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d8073620 con 0x7fe6d8100e60 2024-06-01T00:53:25.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.921+0000 7fe6dc973640 1 -- 172.21.15.79:0/3626813959 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6cc031640 con 0x7fe6d8100e60 2024-06-01T00:53:25.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.921+0000 7fe6dc973640 1 -- 172.21.15.79:0/3626813959 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6cc003820 con 0x7fe6d8100e60 2024-06-01T00:53:25.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.921+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d81a2670 con 0x7fe6d8100e60 2024-06-01T00:53:25.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.922+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d8101990 con 0x7fe6d8100e60 2024-06-01T00:53:25.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.922+0000 7fe6dc973640 1 -- 172.21.15.79:0/3626813959 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe6cc0317b0 con 0x7fe6d8100e60 2024-06-01T00:53:25.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.923+0000 7fe6dc973640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe6c006aef0 0x7fe6c006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:25.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.923+0000 7fe6dc973640 1 -- 172.21.15.79:0/3626813959 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe6cc0aaa60 con 0x7fe6d8100e60 2024-06-01T00:53:25.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.923+0000 7fe6d6ffd640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe6c006aef0 0x7fe6c006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:25.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.923+0000 7fe6d6ffd640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe6c006aef0 0x7fe6c006d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009c20 tx=0x7fe6c8005f90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:25.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:25.927+0000 7fe6dc973640 1 -- 172.21.15.79:0/3626813959 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe6cc076040 con 0x7fe6d8100e60 2024-06-01T00:53:26.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.135+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe6d804e210 con 0x7fe6d8100e60 2024-06-01T00:53:26.136 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.135+0000 7fe6dc973640 1 -- 172.21.15.79:0/3626813959 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe6cc079660 con 0x7fe6d8100e60 2024-06-01T00:53:26.136 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:26.136 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:26.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe6c006aef0 msgr2=0x7fe6c006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:26.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe6c006aef0 0x7fe6c006d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009c20 tx=0x7fe6c8005f90 comp rx=0 tx=0).stop 2024-06-01T00:53:26.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 msgr2=0x7fe6d8072cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:26.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8072cf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc02a9c0 tx=0x7fe6cc0055a0 comp rx=0 tx=0).stop 2024-06-01T00:53:26.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 shutdown_connections 2024-06-01T00:53:26.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe6c006aef0 0x7fe6c006d3b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:26.138 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe6d8100e60 0x7fe6d8072cf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:26.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 --2- 172.21.15.79:0/3626813959 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe6d81004b0 0x7fe6d80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:26.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 >> 172.21.15.79:0/3626813959 conn(0x7fe6d80fbda0 msgr2=0x7fe6d8103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:26.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 shutdown_connections 2024-06-01T00:53:26.139 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:26.137+0000 7fe6dd975640 1 -- 172.21.15.79:0/3626813959 wait complete. 2024-06-01T00:53:26.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:26 smithi079 ceph-mon[29356]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:26.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:26 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3626813959' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:26.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:26 smithi160 ceph-mon[36639]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:26.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:26 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3626813959' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:28.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:28 smithi079 ceph-mon[29356]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:28.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:28 smithi160 ceph-mon[36639]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:29.724 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:29.941 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:30.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:30 smithi160 ceph-mon[36639]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:30.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:30 smithi079 ceph-mon[29356]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:31.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 -- 172.21.15.79:0/2573049549 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 msgr2=0x7fcdd8103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:31.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 --2- 172.21.15.79:0/2573049549 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd8103da0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009880 tx=0x7fcdcc02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 -- 172.21.15.79:0/2573049549 shutdown_connections 2024-06-01T00:53:31.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 --2- 172.21.15.79:0/2573049549 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd8103da0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 --2- 172.21.15.79:0/2573049549 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcdd80ff3a0 0x7fcdd80ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 -- 172.21.15.79:0/2573049549 >> 172.21.15.79:0/2573049549 conn(0x7fcdd80fac50 msgr2=0x7fcdd80fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 -- 172.21.15.79:0/2573049549 shutdown_connections 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.532+0000 7fcdddad6640 1 -- 172.21.15.79:0/2573049549 wait complete. 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdddad6640 1 Processor -- start 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdddad6640 1 -- start start 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdddad6640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcdd80ff3a0 0x7fcdd8195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdddad6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd81957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdddad6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd8195e30 con 0x7fcdd80ffd50 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdddad6640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd8199b50 con 0x7fcdd80ff3a0 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd81957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd81957a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53488/0 (socket says 172.21.15.79:53488) 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.533+0000 7fcdd6ffd640 1 -- 172.21.15.79:0/1970220524 learned_addr learned my addr 172.21.15.79:0/1970220524 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd77fe640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcdd80ff3a0 0x7fcdd8195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd6ffd640 1 -- 172.21.15.79:0/1970220524 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcdd80ff3a0 msgr2=0x7fcdd8195260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd6ffd640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcdd80ff3a0 0x7fcdd8195260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd6ffd640 1 -- 172.21.15.79:0/1970220524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc009560 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd6ffd640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd81957a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc02a9c0 tx=0x7fcdcc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd4ff9640 1 -- 172.21.15.79:0/1970220524 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdcc004aa0 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd8199dd0 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd4ff9640 1 -- 172.21.15.79:0/1970220524 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdcc004c00 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdd4ff9640 1 -- 172.21.15.79:0/1970220524 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdcc003a50 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.534+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd819a2c0 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.535+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd8100880 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.535+0000 7fcdd4ff9640 1 -- 172.21.15.79:0/1970220524 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fcdcc003c90 con 0x7fcdd80ffd50 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.536+0000 7fcdd4ff9640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fcdc006aef0 0x7fcdc006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.536+0000 7fcdd77fe640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fcdc006aef0 0x7fcdc006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:31.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.536+0000 7fcdd4ff9640 1 -- 172.21.15.79:0/1970220524 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fcdcc0aae10 con 0x7fcdd80ffd50 2024-06-01T00:53:31.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.536+0000 7fcdd77fe640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fcdc006aef0 0x7fcdc006d3b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009890 tx=0x7fcdc8008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:31.541 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.540+0000 7fcdd4ff9640 1 -- 172.21.15.79:0/1970220524 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fcdcc076510 con 0x7fcdd80ffd50 2024-06-01T00:53:31.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.747+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcdd804e210 con 0x7fcdd80ffd50 2024-06-01T00:53:31.748 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.747+0000 7fcdd4ff9640 1 -- 172.21.15.79:0/1970220524 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcdcc079b30 con 0x7fcdd80ffd50 2024-06-01T00:53:31.748 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:31.748 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fcdc006aef0 msgr2=0x7fcdc006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fcdc006aef0 0x7fcdc006d3b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009890 tx=0x7fcdc8008040 comp rx=0 tx=0).stop 2024-06-01T00:53:31.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 msgr2=0x7fcdd81957a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd81957a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc02a9c0 tx=0x7fcdcc0055a0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 shutdown_connections 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fcdc006aef0 0x7fcdc006d3b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcdd80ffd50 0x7fcdd81957a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 --2- 172.21.15.79:0/1970220524 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fcdd80ff3a0 0x7fcdd8195260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.749+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 >> 172.21.15.79:0/1970220524 conn(0x7fcdd80fac50 msgr2=0x7fcdd80fc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.750+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 shutdown_connections 2024-06-01T00:53:31.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:31.750+0000 7fcdddad6640 1 -- 172.21.15.79:0/1970220524 wait complete. 2024-06-01T00:53:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:32 smithi160 ceph-mon[36639]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:32 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1970220524' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:32.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:32 smithi079 ceph-mon[29356]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:32.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:32 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1970220524' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:34.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:34 smithi160 ceph-mon[36639]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:34.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:34 smithi079 ceph-mon[29356]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:35.422 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:35.636 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:35.757 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:36.772 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:36 smithi079 ceph-mon[29356]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:36 smithi160 ceph-mon[36639]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:37.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/3595116125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd0107a80 msgr2=0x7f7fd0107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:37.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/3595116125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd0107a80 0x7f7fd0107ee0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80098b0 tx=0x7f7fc802a4d0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/3595116125 shutdown_connections 2024-06-01T00:53:37.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/3595116125 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd0107a80 0x7f7fd0107ee0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/3595116125 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fd006c5f0 0x7f7fd01074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.229 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/3595116125 >> 172.21.15.79:0/3595116125 conn(0x7f7fd006b340 msgr2=0x7f7fd006b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:37.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/3595116125 shutdown_connections 2024-06-01T00:53:37.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.227+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/3595116125 wait complete. 2024-06-01T00:53:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.228+0000 7f7fd4dd9640 1 Processor -- start 2024-06-01T00:53:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.228+0000 7f7fd4dd9640 1 -- start start 2024-06-01T00:53:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fd4dd9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd006c5f0 0x7f7fd01a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fd4dd9640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fd0107a80 0x7f7fd01a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:37.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fd4dd9640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd01a53c0 con 0x7f7fd006c5f0 2024-06-01T00:53:37.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fd4dd9640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd019e9f0 con 0x7f7fd0107a80 2024-06-01T00:53:37.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fce59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd006c5f0 0x7f7fd01a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:37.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fce59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd006c5f0 0x7f7fd01a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:60408/0 (socket says 172.21.15.79:60408) 2024-06-01T00:53:37.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fce59c640 1 -- 172.21.15.79:0/1915384667 learned_addr learned my addr 172.21.15.79:0/1915384667 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:37.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fcdd9b640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fd0107a80 0x7f7fd01a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:37.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fce59c640 1 -- 172.21.15.79:0/1915384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fd0107a80 msgr2=0x7f7fd01a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:37.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fce59c640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fd0107a80 0x7f7fd01a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fce59c640 1 -- 172.21.15.79:0/1915384667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8009560 con 0x7f7fd006c5f0 2024-06-01T00:53:37.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fce59c640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd006c5f0 0x7f7fd01a4860 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d8d0 tx=0x7f7fb800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:37.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fc77fe640 1 -- 172.21.15.79:0/1915384667 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb800bea0 con 0x7f7fd006c5f0 2024-06-01T00:53:37.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fc77fe640 1 -- 172.21.15.79:0/1915384667 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fb8009c00 con 0x7f7fd006c5f0 2024-06-01T00:53:37.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd019ecd0 con 0x7f7fd006c5f0 2024-06-01T00:53:37.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.229+0000 7f7fc77fe640 1 -- 172.21.15.79:0/1915384667 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fb800f710 con 0x7f7fd006c5f0 2024-06-01T00:53:37.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.230+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd019f1f0 con 0x7f7fd006c5f0 2024-06-01T00:53:37.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.230+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd0108a60 con 0x7f7fd006c5f0 2024-06-01T00:53:37.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.231+0000 7f7fc77fe640 1 -- 172.21.15.79:0/1915384667 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7fb8010460 con 0x7f7fd006c5f0 2024-06-01T00:53:37.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.231+0000 7f7fc77fe640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7fbc06aef0 0x7f7fbc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:37.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.231+0000 7f7fc77fe640 1 -- 172.21.15.79:0/1915384667 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7fb80897b0 con 0x7f7fd006c5f0 2024-06-01T00:53:37.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.231+0000 7f7fcdd9b640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7fbc06aef0 0x7f7fbc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:37.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.232+0000 7f7fcdd9b640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7fbc06aef0 0x7f7fbc06d3b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009880 tx=0x7f7fc80058d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:37.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.236+0000 7f7fc77fe640 1 -- 172.21.15.79:0/1915384667 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7fb8054ec0 con 0x7f7fd006c5f0 2024-06-01T00:53:37.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.450+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7fd004e210 con 0x7f7fd006c5f0 2024-06-01T00:53:37.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.450+0000 7f7fc77fe640 1 -- 172.21.15.79:0/1915384667 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7fb80200d0 con 0x7f7fd006c5f0 2024-06-01T00:53:37.452 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:37.452 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:37.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.452+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7fbc06aef0 msgr2=0x7f7fbc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:37.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.452+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7fbc06aef0 0x7f7fbc06d3b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009880 tx=0x7f7fc80058d0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.452+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd006c5f0 msgr2=0x7f7fd01a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.452+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd006c5f0 0x7f7fd01a4860 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7fb800d8d0 tx=0x7f7fb800dd90 comp rx=0 tx=0).stop 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.453+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 shutdown_connections 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.453+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7fbc06aef0 0x7f7fbc06d3b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.453+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7fd0107a80 0x7f7fd01a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.453+0000 7f7fd4dd9640 1 --2- 172.21.15.79:0/1915384667 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fd006c5f0 0x7f7fd01a4860 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.453+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 >> 172.21.15.79:0/1915384667 conn(0x7f7fd006b340 msgr2=0x7f7fd01128d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.453+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 shutdown_connections 2024-06-01T00:53:37.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:37.453+0000 7f7fd4dd9640 1 -- 172.21.15.79:0/1915384667 wait complete. 2024-06-01T00:53:38.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:38 smithi079 ceph-mon[29356]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:38.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:38 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1915384667' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:38 smithi160 ceph-mon[36639]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:38 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1915384667' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:40.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:40 smithi160 ceph-mon[36639]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:40.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:40 smithi079 ceph-mon[29356]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:41.205 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:41.422 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:42.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:42 smithi160 ceph-mon[36639]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:42.730 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:42 smithi079 ceph-mon[29356]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:43.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 -- 172.21.15.79:0/4056949199 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e01004b0 msgr2=0x7f59e0100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:43.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 --2- 172.21.15.79:0/4056949199 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e01004b0 0x7f59e0100890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009880 tx=0x7f59d802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:53:43.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 -- 172.21.15.79:0/4056949199 shutdown_connections 2024-06-01T00:53:43.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 --2- 172.21.15.79:0/4056949199 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59e0100e60 0x7f59e0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:43.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 --2- 172.21.15.79:0/4056949199 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e01004b0 0x7f59e0100890 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:43.089 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 -- 172.21.15.79:0/4056949199 >> 172.21.15.79:0/4056949199 conn(0x7f59e00fbda0 msgr2=0x7f59e00fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:43.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 -- 172.21.15.79:0/4056949199 shutdown_connections 2024-06-01T00:53:43.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.082+0000 7f59e8364640 1 -- 172.21.15.79:0/4056949199 wait complete. 2024-06-01T00:53:43.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e8364640 1 Processor -- start 2024-06-01T00:53:43.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e8364640 1 -- start start 2024-06-01T00:53:43.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e8364640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59e01004b0 0x7f59e0196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:43.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e8364640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e0100e60 0x7f59e01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e8364640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e0196f60 con 0x7f59e0100e60 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e8364640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e019ac80 con 0x7f59e01004b0 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e58ff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e0100e60 0x7f59e01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e58ff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e0100e60 0x7f59e01968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46808/0 (socket says 172.21.15.79:46808) 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.083+0000 7f59e58ff640 1 -- 172.21.15.79:0/3375872451 learned_addr learned my addr 172.21.15.79:0/3375872451 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e6100640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59e01004b0 0x7f59e0196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e58ff640 1 -- 172.21.15.79:0/3375872451 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59e01004b0 msgr2=0x7f59e0196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e58ff640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59e01004b0 0x7f59e0196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e58ff640 1 -- 172.21.15.79:0/3375872451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8009560 con 0x7f59e0100e60 2024-06-01T00:53:43.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e6100640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59e01004b0 0x7f59e0196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e58ff640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e0100e60 0x7f59e01968d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f59d400b3d0 tx=0x7f59d400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59df7fe640 1 -- 172.21.15.79:0/3375872451 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d4011800 con 0x7f59e0100e60 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59df7fe640 1 -- 172.21.15.79:0/3375872451 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d4011e20 con 0x7f59e0100e60 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59df7fe640 1 -- 172.21.15.79:0/3375872451 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59d400f2e0 con 0x7f59e0100e60 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e019af60 con 0x7f59e0100e60 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.084+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e019b4b0 con 0x7f59e0100e60 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.085+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e0101990 con 0x7f59e0100e60 2024-06-01T00:53:43.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.085+0000 7f59df7fe640 1 -- 172.21.15.79:0/3375872451 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f59d4010380 con 0x7f59e0100e60 2024-06-01T00:53:43.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.086+0000 7f59df7fe640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f59c806aef0 0x7f59c806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:43.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.086+0000 7f59df7fe640 1 -- 172.21.15.79:0/3375872451 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f59d4089930 con 0x7f59e0100e60 2024-06-01T00:53:43.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.086+0000 7f59e6100640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f59c806aef0 0x7f59c806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:43.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.086+0000 7f59e6100640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f59c806aef0 0x7f59c806d3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f59d8006010 tx=0x7f59d8005e60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:43.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.090+0000 7f59df7fe640 1 -- 172.21.15.79:0/3375872451 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f59d4054f10 con 0x7f59e0100e60 2024-06-01T00:53:43.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.298+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f59e004e210 con 0x7f59e0100e60 2024-06-01T00:53:43.300 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.298+0000 7f59df7fe640 1 -- 172.21.15.79:0/3375872451 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f59d4058530 con 0x7f59e0100e60 2024-06-01T00:53:43.301 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:43.301 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f59c806aef0 msgr2=0x7f59c806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f59c806aef0 0x7f59c806d3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f59d8006010 tx=0x7f59d8005e60 comp rx=0 tx=0).stop 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e0100e60 msgr2=0x7f59e01968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e0100e60 0x7f59e01968d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f59d400b3d0 tx=0x7f59d400b890 comp rx=0 tx=0).stop 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 shutdown_connections 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f59c806aef0 0x7f59c806d3b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f59e0100e60 0x7f59e01968d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 --2- 172.21.15.79:0/3375872451 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f59e01004b0 0x7f59e0196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 >> 172.21.15.79:0/3375872451 conn(0x7f59e00fbda0 msgr2=0x7f59e00fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 shutdown_connections 2024-06-01T00:53:43.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:43.301+0000 7f59e8364640 1 -- 172.21.15.79:0/3375872451 wait complete. 2024-06-01T00:53:43.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:43 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3375872451' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:43.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:43 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3375872451' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:44 smithi160 ceph-mon[36639]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:44.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:44 smithi079 ceph-mon[29356]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:46.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:46 smithi160 ceph-mon[36639]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:46.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:46 smithi079 ceph-mon[29356]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:46.907 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:47.126 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:48.726 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:48 smithi079 ceph-mon[29356]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:48.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:48 smithi160 ceph-mon[36639]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 -- 172.21.15.79:0/1643791035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101e40 msgr2=0x7f0e4410a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 --2- 172.21.15.79:0/1643791035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101e40 0x7f0e4410a330 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0098b0 tx=0x7f0e2c02a560 comp rx=0 tx=0).stop 2024-06-01T00:53:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 -- 172.21.15.79:0/1643791035 shutdown_connections 2024-06-01T00:53:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 --2- 172.21.15.79:0/1643791035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101e40 0x7f0e4410a330 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 --2- 172.21.15.79:0/1643791035 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e44101520 0x7f0e44101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:48.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 -- 172.21.15.79:0/1643791035 >> 172.21.15.79:0/1643791035 conn(0x7f0e440facf0 msgr2=0x7f0e440fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 -- 172.21.15.79:0/1643791035 shutdown_connections 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.938+0000 7f0e49657640 1 -- 172.21.15.79:0/1643791035 wait complete. 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e49657640 1 Processor -- start 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e49657640 1 -- start start 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e49657640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101520 0x7f0e44196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e49657640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e44101e40 0x7f0e44196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e49657640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e44196f00 con 0x7f0e44101520 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e49657640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4419ac20 con 0x7f0e44101e40 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e42ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101520 0x7f0e44196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:48.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e42ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101520 0x7f0e44196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46822/0 (socket says 172.21.15.79:46822) 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.939+0000 7f0e42ffd640 1 -- 172.21.15.79:0/2731959967 learned_addr learned my addr 172.21.15.79:0/2731959967 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e3a7fc640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e44101e40 0x7f0e44196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e42ffd640 1 -- 172.21.15.79:0/2731959967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e44101e40 msgr2=0x7f0e44196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e42ffd640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e44101e40 0x7f0e44196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e42ffd640 1 -- 172.21.15.79:0/2731959967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c009560 con 0x7f0e44101520 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e42ffd640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101520 0x7f0e44196330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400b420 tx=0x7f0e3400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e40ff9640 1 -- 172.21.15.79:0/2731959967 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e34011800 con 0x7f0e44101520 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e40ff9640 1 -- 172.21.15.79:0/2731959967 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e34011e20 con 0x7f0e44101520 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4419af00 con 0x7f0e44101520 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e40ff9640 1 -- 172.21.15.79:0/2731959967 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e3400f2e0 con 0x7f0e44101520 2024-06-01T00:53:48.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.940+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4419b420 con 0x7f0e44101520 2024-06-01T00:53:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.941+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e44068f60 con 0x7f0e44101520 2024-06-01T00:53:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.942+0000 7f0e40ff9640 1 -- 172.21.15.79:0/2731959967 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0e34010380 con 0x7f0e44101520 2024-06-01T00:53:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.942+0000 7f0e40ff9640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e3006aef0 0x7f0e3006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.942+0000 7f0e40ff9640 1 -- 172.21.15.79:0/2731959967 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0e34089570 con 0x7f0e44101520 2024-06-01T00:53:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.942+0000 7f0e3a7fc640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e3006aef0 0x7f0e3006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:48.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.943+0000 7f0e3a7fc640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e3006aef0 0x7f0e3006d3b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009230 tx=0x7f0e2c005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:48.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:48.946+0000 7f0e40ff9640 1 -- 172.21.15.79:0/2731959967 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0e34054b50 con 0x7f0e44101520 2024-06-01T00:53:49.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.154+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e4404e210 con 0x7f0e44101520 2024-06-01T00:53:49.155 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.155+0000 7f0e40ff9640 1 -- 172.21.15.79:0/2731959967 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0e34058170 con 0x7f0e44101520 2024-06-01T00:53:49.156 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:49.156 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:49.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.156+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e3006aef0 msgr2=0x7f0e3006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:49.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.156+0000 7f0e49657640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e3006aef0 0x7f0e3006d3b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009230 tx=0x7f0e2c005a20 comp rx=0 tx=0).stop 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.156+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101520 msgr2=0x7f0e44196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.156+0000 7f0e49657640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101520 0x7f0e44196330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400b420 tx=0x7f0e3400b8e0 comp rx=0 tx=0).stop 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.157+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 shutdown_connections 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.157+0000 7f0e49657640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e3006aef0 0x7f0e3006d3b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.157+0000 7f0e49657640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e44101e40 0x7f0e44196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.157+0000 7f0e49657640 1 --2- 172.21.15.79:0/2731959967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e44101520 0x7f0e44196330 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.157+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 >> 172.21.15.79:0/2731959967 conn(0x7f0e440facf0 msgr2=0x7f0e44102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.157+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 shutdown_connections 2024-06-01T00:53:49.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:49.157+0000 7f0e49657640 1 -- 172.21.15.79:0/2731959967 wait complete. 2024-06-01T00:53:49.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:49 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2731959967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:49.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:49 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2731959967' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:50 smithi160 ceph-mon[36639]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:50.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:50.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:50 smithi079 ceph-mon[29356]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:53:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:52 smithi160 ceph-mon[36639]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:52.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:52 smithi079 ceph-mon[29356]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:52.897 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:53.112 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:53:54.577 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:54 smithi079 ceph-mon[29356]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:54.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 -- 172.21.15.79:0/2849693946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 msgr2=0x7f6b6c0ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:54.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 --2- 172.21.15.79:0/2849693946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c0ff800 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009880 tx=0x7f6b5c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 -- 172.21.15.79:0/2849693946 shutdown_connections 2024-06-01T00:53:54.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 --2- 172.21.15.79:0/2849693946 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b6c0ffdd0 0x7f6b6c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 --2- 172.21.15.79:0/2849693946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c0ff800 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.794 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 -- 172.21.15.79:0/2849693946 >> 172.21.15.79:0/2849693946 conn(0x7f6b6c0facf0 msgr2=0x7f6b6c0fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:54.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 -- 172.21.15.79:0/2849693946 shutdown_connections 2024-06-01T00:53:54.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 -- 172.21.15.79:0/2849693946 wait complete. 2024-06-01T00:53:54.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.780+0000 7f6b71892640 1 Processor -- start 2024-06-01T00:53:54.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b71892640 1 -- start start 2024-06-01T00:53:54.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b71892640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:54.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b71892640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b6c0ffdd0 0x7f6b6c196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:54.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b71892640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c196ed0 con 0x7f6b6c0ff420 2024-06-01T00:53:54.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b71892640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c19abf0 con 0x7f6b6c0ffdd0 2024-06-01T00:53:54.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b6affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:54.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b6affd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52478/0 (socket says 172.21.15.79:52478) 2024-06-01T00:53:54.796 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b6affd640 1 -- 172.21.15.79:0/385952305 learned_addr learned my addr 172.21.15.79:0/385952305 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b6affd640 1 -- 172.21.15.79:0/385952305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b6c0ffdd0 msgr2=0x7f6b6c196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b6affd640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b6c0ffdd0 0x7f6b6c196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b6affd640 1 -- 172.21.15.79:0/385952305 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c009560 con 0x7f6b6c0ff420 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.781+0000 7f6b6affd640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c196300 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c02a9c0 tx=0x7f6b5c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.782+0000 7f6b70890640 1 -- 172.21.15.79:0/385952305 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b5c004aa0 con 0x7f6b6c0ff420 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.782+0000 7f6b70890640 1 -- 172.21.15.79:0/385952305 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b5c004c00 con 0x7f6b6c0ff420 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.782+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c19ae70 con 0x7f6b6c0ff420 2024-06-01T00:53:54.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.782+0000 7f6b70890640 1 -- 172.21.15.79:0/385952305 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b5c003ab0 con 0x7f6b6c0ff420 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.782+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c19b360 con 0x7f6b6c0ff420 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.783+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c100900 con 0x7f6b6c0ff420 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.783+0000 7f6b70890640 1 -- 172.21.15.79:0/385952305 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f6b5c031d20 con 0x7f6b6c0ff420 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.783+0000 7f6b70890640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6b5806aef0 0x7f6b5806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.784+0000 7f6b70890640 1 -- 172.21.15.79:0/385952305 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6b5c0aad10 con 0x7f6b6c0ff420 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.784+0000 7f6b6a7fc640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6b5806aef0 0x7f6b5806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.784+0000 7f6b6a7fc640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6b5806aef0 0x7f6b5806d3b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c197860 tx=0x7f6b54006c90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:53:54.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.788+0000 7f6b70890640 1 -- 172.21.15.79:0/385952305 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f6b5c0762f0 con 0x7f6b6c0ff420 2024-06-01T00:53:54.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:54 smithi160 ceph-mon[36639]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:54.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.994+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6b6c04e210 con 0x7f6b6c0ff420 2024-06-01T00:53:54.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.994+0000 7f6b70890640 1 -- 172.21.15.79:0/385952305 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6b5c079910 con 0x7f6b6c0ff420 2024-06-01T00:53:54.995 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:53:54.995 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:53:54.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6b5806aef0 msgr2=0x7f6b5806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6b5806aef0 0x7f6b5806d3b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c197860 tx=0x7f6b54006c90 comp rx=0 tx=0).stop 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 msgr2=0x7f6b6c196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c196300 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c02a9c0 tx=0x7f6b5c0055a0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 shutdown_connections 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f6b5806aef0 0x7f6b5806d3b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f6b6c0ffdd0 0x7f6b6c196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 --2- 172.21.15.79:0/385952305 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b6c0ff420 0x7f6b6c196300 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:53:54.998 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 >> 172.21.15.79:0/385952305 conn(0x7f6b6c0facf0 msgr2=0x7f6b6c068a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:53:54.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 shutdown_connections 2024-06-01T00:53:54.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:53:54.996+0000 7f6b71892640 1 -- 172.21.15.79:0/385952305 wait complete. 2024-06-01T00:53:55.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:55 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/385952305' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:55.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:55 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/385952305' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:53:56.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:56 smithi160 ceph-mon[36639]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:56.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:56 smithi079 ceph-mon[29356]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:58.687 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:53:58.712 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:53:58 smithi079 ceph-mon[29356]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:58.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:53:58 smithi160 ceph-mon[36639]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:53:58.897 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 -- 172.21.15.79:0/4176892262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 msgr2=0x7f300c1017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 --2- 172.21.15.79:0/4176892262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c1017f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009880 tx=0x7f2ff402a530 comp rx=0 tx=0).stop 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 -- 172.21.15.79:0/4176892262 shutdown_connections 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 --2- 172.21.15.79:0/4176892262 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f300c101d30 0x7f300c10a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 --2- 172.21.15.79:0/4176892262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c1017f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 -- 172.21.15.79:0/4176892262 >> 172.21.15.79:0/4176892262 conn(0x7f300c0fac80 msgr2=0x7f300c0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 -- 172.21.15.79:0/4176892262 shutdown_connections 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.502+0000 7f3010e69640 1 -- 172.21.15.79:0/4176892262 wait complete. 2024-06-01T00:54:00.508 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f3010e69640 1 Processor -- start 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f3010e69640 1 -- start start 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f3010e69640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f3010e69640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f300c101d30 0x7f300c196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f3010e69640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c196ee0 con 0x7f300c101410 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f3010e69640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c19ac00 con 0x7f300c101d30 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f300a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f300a59c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52502/0 (socket says 172.21.15.79:52502) 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.503+0000 7f300a59c640 1 -- 172.21.15.79:0/2294309929 learned_addr learned my addr 172.21.15.79:0/2294309929 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f3009d9b640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f300c101d30 0x7f300c196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f300a59c640 1 -- 172.21.15.79:0/2294309929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f300c101d30 msgr2=0x7f300c196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f300a59c640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f300c101d30 0x7f300c196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f300a59c640 1 -- 172.21.15.79:0/2294309929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4009560 con 0x7f300c101410 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f3009d9b640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f300c101d30 0x7f300c196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:54:00.509 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f300a59c640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c196310 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4006010 tx=0x7f2ff40055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f30037fe640 1 -- 172.21.15.79:0/2294309929 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ff4004aa0 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f30037fe640 1 -- 172.21.15.79:0/2294309929 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ff4004c00 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c19ae80 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f30037fe640 1 -- 172.21.15.79:0/2294309929 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ff4003a20 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.504+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c19b370 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.505+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c069060 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.506+0000 7f30037fe640 1 -- 172.21.15.79:0/2294309929 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2ff402ad50 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.506+0000 7f30037fe640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2ff806af00 0x7f2ff806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.506+0000 7f30037fe640 1 -- 172.21.15.79:0/2294309929 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2ff40aab80 con 0x7f300c101410 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.506+0000 7f3009d9b640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2ff806af00 0x7f2ff806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:00.510 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.507+0000 7f3009d9b640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2ff806af00 0x7f2ff806d3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc006fd0 tx=0x7f2ffc008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:00.511 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.510+0000 7f30037fe640 1 -- 172.21.15.79:0/2294309929 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2ff4076160 con 0x7f300c101410 2024-06-01T00:54:00.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:00 smithi079 ceph-mon[29356]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:00.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.722+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f300c04e210 con 0x7f300c101410 2024-06-01T00:54:00.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.722+0000 7f30037fe640 1 -- 172.21.15.79:0/2294309929 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2ff4079780 con 0x7f300c101410 2024-06-01T00:54:00.723 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:00.724 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:00.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2ff806af00 msgr2=0x7f2ff806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:00.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2ff806af00 0x7f2ff806d3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc006fd0 tx=0x7f2ffc008040 comp rx=0 tx=0).stop 2024-06-01T00:54:00.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 msgr2=0x7f300c196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:00.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c196310 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4006010 tx=0x7f2ff40055a0 comp rx=0 tx=0).stop 2024-06-01T00:54:00.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 shutdown_connections 2024-06-01T00:54:00.725 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f2ff806af00 0x7f2ff806d3c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:00.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f300c101d30 0x7f300c196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:00.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 --2- 172.21.15.79:0/2294309929 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f300c101410 0x7f300c196310 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:00.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 >> 172.21.15.79:0/2294309929 conn(0x7f300c0fac80 msgr2=0x7f300c102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:00.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 shutdown_connections 2024-06-01T00:54:00.726 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:00.724+0000 7f3010e69640 1 -- 172.21.15.79:0/2294309929 wait complete. 2024-06-01T00:54:00.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:00 smithi160 ceph-mon[36639]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:01.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:01 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2294309929' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:01.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:01 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2294309929' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:02.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:02 smithi160 ceph-mon[36639]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:02.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:02 smithi079 ceph-mon[29356]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:04.493 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:04.707 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:04.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:04 smithi160 ceph-mon[36639]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:04.828 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:04 smithi079 ceph-mon[29356]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:05.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:05.846 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:06.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.386+0000 7fd4ce11e640 1 -- 172.21.15.79:0/3679186575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 msgr2=0x7fd4c8103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:06.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.386+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/3679186575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c8103d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009880 tx=0x7fd4bc02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.386+0000 7fd4ce11e640 1 -- 172.21.15.79:0/3679186575 shutdown_connections 2024-06-01T00:54:06.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.386+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/3679186575 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c8103d60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.386+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/3679186575 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c80ff360 0x7fd4c80ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.386+0000 7fd4ce11e640 1 -- 172.21.15.79:0/3679186575 >> 172.21.15.79:0/3679186575 conn(0x7fd4c8075e10 msgr2=0x7fd4c80fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:06.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.386+0000 7fd4ce11e640 1 -- 172.21.15.79:0/3679186575 shutdown_connections 2024-06-01T00:54:06.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.387+0000 7fd4ce11e640 1 -- 172.21.15.79:0/3679186575 wait complete. 2024-06-01T00:54:06.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.387+0000 7fd4ce11e640 1 Processor -- start 2024-06-01T00:54:06.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.387+0000 7fd4ce11e640 1 -- start start 2024-06-01T00:54:06.389 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4ce11e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c80ff360 0x7fd4c8197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4ce11e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c81975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4ce11e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c8197af0 con 0x7fd4c80ffd10 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4ce11e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c8197c60 con 0x7fd4c80ff360 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c81975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c6ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c81975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49920/0 (socket says 172.21.15.79:49920) 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c6ffd640 1 -- 172.21.15.79:0/2376957823 learned_addr learned my addr 172.21.15.79:0/2376957823 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c77fe640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c80ff360 0x7fd4c8197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c6ffd640 1 -- 172.21.15.79:0/2376957823 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c80ff360 msgr2=0x7fd4c8197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c6ffd640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c80ff360 0x7fd4c8197070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c6ffd640 1 -- 172.21.15.79:0/2376957823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc009560 con 0x7fd4c80ffd10 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.388+0000 7fd4c6ffd640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c81975b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc02a9c0 tx=0x7fd4bc0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.389+0000 7fd4c4ff9640 1 -- 172.21.15.79:0/2376957823 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4bc004aa0 con 0x7fd4c80ffd10 2024-06-01T00:54:06.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.389+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c8197ee0 con 0x7fd4c80ffd10 2024-06-01T00:54:06.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.389+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c81a0410 con 0x7fd4c80ffd10 2024-06-01T00:54:06.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.389+0000 7fd4c4ff9640 1 -- 172.21.15.79:0/2376957823 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4bc031470 con 0x7fd4c80ffd10 2024-06-01T00:54:06.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.389+0000 7fd4c4ff9640 1 -- 172.21.15.79:0/2376957823 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4bc003940 con 0x7fd4c80ffd10 2024-06-01T00:54:06.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.390+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b4005250 con 0x7fd4c80ffd10 2024-06-01T00:54:06.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.390+0000 7fd4c4ff9640 1 -- 172.21.15.79:0/2376957823 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd4bc004c00 con 0x7fd4c80ffd10 2024-06-01T00:54:06.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.391+0000 7fd4c4ff9640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd4b006aef0 0x7fd4b006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:06.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.391+0000 7fd4c4ff9640 1 -- 172.21.15.79:0/2376957823 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd4bc0aa930 con 0x7fd4c80ffd10 2024-06-01T00:54:06.392 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.391+0000 7fd4c77fe640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd4b006aef0 0x7fd4b006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:06.393 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.392+0000 7fd4c77fe640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd4b006aef0 0x7fd4b006d3b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8005d00 tx=0x7fd4b800b410 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:06.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.396+0000 7fd4c4ff9640 1 -- 172.21.15.79:0/2376957823 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd4bc0760e0 con 0x7fd4c80ffd10 2024-06-01T00:54:06.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.615+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd4b4005a80 con 0x7fd4c80ffd10 2024-06-01T00:54:06.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.616+0000 7fd4c4ff9640 1 -- 172.21.15.79:0/2376957823 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd4bc079700 con 0x7fd4c80ffd10 2024-06-01T00:54:06.617 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:06.617 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd4b006aef0 msgr2=0x7fd4b006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd4b006aef0 0x7fd4b006d3b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8005d00 tx=0x7fd4b800b410 comp rx=0 tx=0).stop 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 msgr2=0x7fd4c81975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c81975b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc02a9c0 tx=0x7fd4bc0055a0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 shutdown_connections 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd4b006aef0 0x7fd4b006d3b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4c80ffd10 0x7fd4c81975b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 --2- 172.21.15.79:0/2376957823 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd4c80ff360 0x7fd4c8197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 >> 172.21.15.79:0/2376957823 conn(0x7fd4c8075e10 msgr2=0x7fd4c8102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:06.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 shutdown_connections 2024-06-01T00:54:06.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:06.618+0000 7fd4ce11e640 1 -- 172.21.15.79:0/2376957823 wait complete. 2024-06-01T00:54:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:06 smithi160 ceph-mon[36639]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:06 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:54:06.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:06 smithi079 ceph-mon[29356]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:06.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:06 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:54:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:07 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2376957823' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:54:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:54:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:54:07.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:07 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2376957823' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:07.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:54:07.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:54:07.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:54:08.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:08 smithi160 ceph-mon[36639]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:08.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:08 smithi079 ceph-mon[29356]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:10.382 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:10.598 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:10.721 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:10 smithi079 ceph-mon[29356]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:10.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:10 smithi160 ceph-mon[36639]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.247+0000 7f3db5fa7640 1 -- 172.21.15.79:0/496283573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 msgr2=0x7f3db00ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.247+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/496283573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db00ff770 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009880 tx=0x7f3da002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 -- 172.21.15.79:0/496283573 shutdown_connections 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/496283573 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3db00ffd40 0x7f3db0103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/496283573 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db00ff770 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 -- 172.21.15.79:0/496283573 >> 172.21.15.79:0/496283573 conn(0x7f3db00fac40 msgr2=0x7f3db00fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 -- 172.21.15.79:0/496283573 shutdown_connections 2024-06-01T00:54:12.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 -- 172.21.15.79:0/496283573 wait complete. 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 Processor -- start 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 -- start start 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3db00ffd40 0x7f3db0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db0199140 con 0x7f3db00ff390 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3db5fa7640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db019ce10 con 0x7f3db00ffd40 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.248+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3daf7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db0198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50892/0 (socket says 172.21.15.79:50892) 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3daf7fe640 1 -- 172.21.15.79:0/3362007316 learned_addr learned my addr 172.21.15.79:0/3362007316 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3da7fff640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3db00ffd40 0x7f3db0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3daf7fe640 1 -- 172.21.15.79:0/3362007316 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3db00ffd40 msgr2=0x7f3db0198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3daf7fe640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3db00ffd40 0x7f3db0198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3daf7fe640 1 -- 172.21.15.79:0/3362007316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da0009560 con 0x7f3db00ff390 2024-06-01T00:54:12.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3daf7fe640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db0198570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3da002a9c0 tx=0x7f3da00055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:12.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3dad7fa640 1 -- 172.21.15.79:0/3362007316 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3da0004aa0 con 0x7f3db00ff390 2024-06-01T00:54:12.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db019d090 con 0x7f3db00ff390 2024-06-01T00:54:12.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3dad7fa640 1 -- 172.21.15.79:0/3362007316 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3da0004c00 con 0x7f3db00ff390 2024-06-01T00:54:12.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3dad7fa640 1 -- 172.21.15.79:0/3362007316 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3da0003a20 con 0x7f3db00ff390 2024-06-01T00:54:12.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.249+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db019d580 con 0x7f3db00ff390 2024-06-01T00:54:12.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.250+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db0100870 con 0x7f3db00ff390 2024-06-01T00:54:12.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.251+0000 7f3dad7fa640 1 -- 172.21.15.79:0/3362007316 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f3da0003b80 con 0x7f3db00ff390 2024-06-01T00:54:12.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.251+0000 7f3dad7fa640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3d9c06aef0 0x7f3d9c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:12.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.251+0000 7f3dad7fa640 1 -- 172.21.15.79:0/3362007316 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3da00aad30 con 0x7f3db00ff390 2024-06-01T00:54:12.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.251+0000 7f3da7fff640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3d9c06aef0 0x7f3d9c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:12.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.251+0000 7f3da7fff640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3d9c06aef0 0x7f3d9c06d3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3db0199a80 tx=0x7f3d98009110 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:12.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.255+0000 7f3dad7fa640 1 -- 172.21.15.79:0/3362007316 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f3da0076310 con 0x7f3db00ff390 2024-06-01T00:54:12.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.466+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3db01998c0 con 0x7f3db00ff390 2024-06-01T00:54:12.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.466+0000 7f3dad7fa640 1 -- 172.21.15.79:0/3362007316 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3da0079930 con 0x7f3db00ff390 2024-06-01T00:54:12.467 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:12.467 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3d9c06aef0 msgr2=0x7f3d9c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3d9c06aef0 0x7f3d9c06d3b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3db0199a80 tx=0x7f3d98009110 comp rx=0 tx=0).stop 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 msgr2=0x7f3db0198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db0198570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3da002a9c0 tx=0x7f3da00055a0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 shutdown_connections 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f3d9c06aef0 0x7f3d9c06d3b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f3db00ffd40 0x7f3db0198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 --2- 172.21.15.79:0/3362007316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3db00ff390 0x7f3db0198570 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:12.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 >> 172.21.15.79:0/3362007316 conn(0x7f3db00fac40 msgr2=0x7f3db0068d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:12.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 shutdown_connections 2024-06-01T00:54:12.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:12.468+0000 7f3db5fa7640 1 -- 172.21.15.79:0/3362007316 wait complete. 2024-06-01T00:54:12.781 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:12 smithi079 ceph-mon[29356]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:12.781 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:12 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3362007316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:12 smithi160 ceph-mon[36639]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:12 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3362007316' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:14.633 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:14 smithi079 ceph-mon[29356]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:14.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:14 smithi160 ceph-mon[36639]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:16.082 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:16.292 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:16.717 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:16 smithi079 ceph-mon[29356]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:16.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:16 smithi160 ceph-mon[36639]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 -- 172.21.15.79:0/579281256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ffb20 msgr2=0x7f0e38103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/579281256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ffb20 0x7f0e38103ff0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009880 tx=0x7f0e2002a530 comp rx=0 tx=0).stop 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 -- 172.21.15.79:0/579281256 shutdown_connections 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/579281256 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ffb20 0x7f0e38103ff0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/579281256 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e380ff200 0x7f0e380ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 -- 172.21.15.79:0/579281256 >> 172.21.15.79:0/579281256 conn(0x7f0e380fac80 msgr2=0x7f0e380fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 -- 172.21.15.79:0/579281256 shutdown_connections 2024-06-01T00:54:17.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.827+0000 7f0e3d66d640 1 -- 172.21.15.79:0/579281256 wait complete. 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e3d66d640 1 Processor -- start 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e3d66d640 1 -- start start 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e3d66d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ff200 0x7f0e381a2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e3d66d640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e380ffb20 0x7f0e381a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e3d66d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3819c7a0 con 0x7f0e380ff200 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e3d66d640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3819c910 con 0x7f0e380ffb20 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e36ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ff200 0x7f0e381a2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e36ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ff200 0x7f0e381a2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50912/0 (socket says 172.21.15.79:50912) 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e36ffd640 1 -- 172.21.15.79:0/2847172911 learned_addr learned my addr 172.21.15.79:0/2847172911 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:17.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e36ffd640 1 -- 172.21.15.79:0/2847172911 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e380ffb20 msgr2=0x7f0e381a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e36ffd640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e380ffb20 0x7f0e381a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e36ffd640 1 -- 172.21.15.79:0/2847172911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e20009560 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.828+0000 7f0e36ffd640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ff200 0x7f0e381a2660 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000b420 tx=0x7f0e3000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.829+0000 7f0e2ffff640 1 -- 172.21.15.79:0/2847172911 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e30011800 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.829+0000 7f0e2ffff640 1 -- 172.21.15.79:0/2847172911 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e30011e20 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.829+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3819cbf0 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.829+0000 7f0e2ffff640 1 -- 172.21.15.79:0/2847172911 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e3000f2e0 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.829+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3819d140 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.830+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e38069060 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.830+0000 7f0e2ffff640 1 -- 172.21.15.79:0/2847172911 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0e30010380 con 0x7f0e380ff200 2024-06-01T00:54:17.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.831+0000 7f0e2ffff640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e2406aef0 0x7f0e2406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:17.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.831+0000 7f0e367fc640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e2406aef0 0x7f0e2406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:17.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.831+0000 7f0e2ffff640 1 -- 172.21.15.79:0/2847172911 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0e30089570 con 0x7f0e380ff200 2024-06-01T00:54:17.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.831+0000 7f0e367fc640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e2406aef0 0x7f0e2406d3b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0e3819dbc0 tx=0x7f0e20000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:17.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:17.835+0000 7f0e2ffff640 1 -- 172.21.15.79:0/2847172911 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0e30054b50 con 0x7f0e380ff200 2024-06-01T00:54:18.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.042+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e3819d3c0 con 0x7f0e380ff200 2024-06-01T00:54:18.043 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.042+0000 7f0e2ffff640 1 -- 172.21.15.79:0/2847172911 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0e30058170 con 0x7f0e380ff200 2024-06-01T00:54:18.043 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:18.043 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e2406aef0 msgr2=0x7f0e2406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e2406aef0 0x7f0e2406d3b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0e3819dbc0 tx=0x7f0e20000bc0 comp rx=0 tx=0).stop 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ff200 msgr2=0x7f0e381a2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ff200 0x7f0e381a2660 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000b420 tx=0x7f0e3000b8e0 comp rx=0 tx=0).stop 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 shutdown_connections 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0e2406aef0 0x7f0e2406d3b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0e380ffb20 0x7f0e381a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 --2- 172.21.15.79:0/2847172911 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e380ff200 0x7f0e381a2660 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 >> 172.21.15.79:0/2847172911 conn(0x7f0e380fac80 msgr2=0x7f0e3810ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 shutdown_connections 2024-06-01T00:54:18.045 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:18.044+0000 7f0e3d66d640 1 -- 172.21.15.79:0/2847172911 wait complete. 2024-06-01T00:54:18.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:18 smithi079 ceph-mon[29356]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:18.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:18 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2847172911' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:18 smithi160 ceph-mon[36639]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:18 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2847172911' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:20 smithi160 ceph-mon[36639]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:20.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:20 smithi079 ceph-mon[29356]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:20.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:21.683 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:21.899 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:22.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:22 smithi079 ceph-mon[29356]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:22.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:22 smithi160 ceph-mon[36639]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:23.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 -- 172.21.15.79:0/1175065589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 msgr2=0x7f9f84100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:23.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/1175065589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84100850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009880 tx=0x7f9f6c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 -- 172.21.15.79:0/1175065589 shutdown_connections 2024-06-01T00:54:23.517 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/1175065589 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9f84100e20 0x7f9f84104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/1175065589 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84100850 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 -- 172.21.15.79:0/1175065589 >> 172.21.15.79:0/1175065589 conn(0x7f9f840fbd20 msgr2=0x7f9f840fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:23.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 -- 172.21.15.79:0/1175065589 shutdown_connections 2024-06-01T00:54:23.518 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.515+0000 7f9f89b9e640 1 -- 172.21.15.79:0/1175065589 wait complete. 2024-06-01T00:54:23.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.516+0000 7f9f89b9e640 1 Processor -- start 2024-06-01T00:54:23.519 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.516+0000 7f9f89b9e640 1 -- start start 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.516+0000 7f9f89b9e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f89b9e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9f84100e20 0x7f9f84196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f89b9e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f84196f00 con 0x7f9f84100470 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f89b9e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8419ac20 con 0x7f9f84100e20 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f837fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35566/0 (socket says 172.21.15.79:35566) 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f837fe640 1 -- 172.21.15.79:0/3671822366 learned_addr learned my addr 172.21.15.79:0/3671822366 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f82ffd640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9f84100e20 0x7f9f84196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f837fe640 1 -- 172.21.15.79:0/3671822366 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9f84100e20 msgr2=0x7f9f84196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f837fe640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9f84100e20 0x7f9f84196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f837fe640 1 -- 172.21.15.79:0/3671822366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c009560 con 0x7f9f84100470 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f82ffd640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9f84100e20 0x7f9f84196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:54:23.520 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f837fe640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84196330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c02a9c0 tx=0x7f9f6c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:23.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f80ff9640 1 -- 172.21.15.79:0/3671822366 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f6c004aa0 con 0x7f9f84100470 2024-06-01T00:54:23.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f80ff9640 1 -- 172.21.15.79:0/3671822366 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f6c004c00 con 0x7f9f84100470 2024-06-01T00:54:23.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8419aea0 con 0x7f9f84100470 2024-06-01T00:54:23.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f80ff9640 1 -- 172.21.15.79:0/3671822366 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f6c003ab0 con 0x7f9f84100470 2024-06-01T00:54:23.521 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.517+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8419b390 con 0x7f9f84100470 2024-06-01T00:54:23.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.518+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f84101950 con 0x7f9f84100470 2024-06-01T00:54:23.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.519+0000 7f9f80ff9640 1 -- 172.21.15.79:0/3671822366 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f9f6c003c10 con 0x7f9f84100470 2024-06-01T00:54:23.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.519+0000 7f9f80ff9640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f9f7006aef0 0x7f9f7006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:23.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.519+0000 7f9f80ff9640 1 -- 172.21.15.79:0/3671822366 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9f6c0aac90 con 0x7f9f84100470 2024-06-01T00:54:23.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.519+0000 7f9f82ffd640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f9f7006aef0 0x7f9f7006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:23.522 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.520+0000 7f9f82ffd640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f9f7006aef0 0x7f9f7006d3b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f84197890 tx=0x7f9f74008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:23.525 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.523+0000 7f9f80ff9640 1 -- 172.21.15.79:0/3671822366 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9f6c0762f0 con 0x7f9f84100470 2024-06-01T00:54:23.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.747+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9f8404e210 con 0x7f9f84100470 2024-06-01T00:54:23.750 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.747+0000 7f9f80ff9640 1 -- 172.21.15.79:0/3671822366 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9f6c079910 con 0x7f9f84100470 2024-06-01T00:54:23.750 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:23.750 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:23.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.749+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f9f7006aef0 msgr2=0x7f9f7006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:23.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.749+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f9f7006aef0 0x7f9f7006d3b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9f84197890 tx=0x7f9f74008040 comp rx=0 tx=0).stop 2024-06-01T00:54:23.751 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 msgr2=0x7f9f84196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84196330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c02a9c0 tx=0x7f9f6c0055a0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 shutdown_connections 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f9f7006aef0 0x7f9f7006d3b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f9f84100e20 0x7f9f84196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 --2- 172.21.15.79:0/3671822366 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9f84100470 0x7f9f84196330 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 >> 172.21.15.79:0/3671822366 conn(0x7f9f840fbd20 msgr2=0x7f9f840fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 shutdown_connections 2024-06-01T00:54:23.752 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:23.750+0000 7f9f89b9e640 1 -- 172.21.15.79:0/3671822366 wait complete. 2024-06-01T00:54:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:24 smithi160 ceph-mon[36639]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:24 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3671822366' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:24.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:24 smithi079 ceph-mon[29356]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:24.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:24 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3671822366' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:26.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:26 smithi160 ceph-mon[36639]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:26.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:26 smithi079 ceph-mon[29356]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:27.487 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:27.703 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:28.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:28 smithi160 ceph-mon[36639]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:28.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:28 smithi079 ceph-mon[29356]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 -- 172.21.15.79:0/3147470303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ffd70 msgr2=0x7fb020103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 --2- 172.21.15.79:0/3147470303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ffd70 0x7fb020103dc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb010009880 tx=0x7fb01002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 -- 172.21.15.79:0/3147470303 shutdown_connections 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 --2- 172.21.15.79:0/3147470303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ffd70 0x7fb020103dc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 --2- 172.21.15.79:0/3147470303 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0200ff3c0 0x7fb0200ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 -- 172.21.15.79:0/3147470303 >> 172.21.15.79:0/3147470303 conn(0x7fb0200fac90 msgr2=0x7fb0200fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 -- 172.21.15.79:0/3147470303 shutdown_connections 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 -- 172.21.15.79:0/3147470303 wait complete. 2024-06-01T00:54:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.362+0000 7fb027541640 1 Processor -- start 2024-06-01T00:54:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb027541640 1 -- start start 2024-06-01T00:54:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb027541640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ff3c0 0x7fb020197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb027541640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0200ffd70 0x7fb0201975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb027541640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020197b10 con 0x7fb0200ff3c0 2024-06-01T00:54:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb027541640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020197c80 con 0x7fb0200ffd70 2024-06-01T00:54:29.366 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0252dd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ff3c0 0x7fb020197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0252dd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ff3c0 0x7fb020197090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35574/0 (socket says 172.21.15.79:35574) 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0252dd640 1 -- 172.21.15.79:0/2380220640 learned_addr learned my addr 172.21.15.79:0/2380220640 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb024adc640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0200ffd70 0x7fb0201975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0252dd640 1 -- 172.21.15.79:0/2380220640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0200ffd70 msgr2=0x7fb0201975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0252dd640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0200ffd70 0x7fb0201975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0252dd640 1 -- 172.21.15.79:0/2380220640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb010009560 con 0x7fb0200ff3c0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb024adc640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0200ffd70 0x7fb0201975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0252dd640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ff3c0 0x7fb020197090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c8a0 tx=0x7fb01c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0167fc640 1 -- 172.21.15.79:0/2380220640 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb01c007b80 con 0x7fb0200ff3c0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0167fc640 1 -- 172.21.15.79:0/2380220640 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb01c007ce0 con 0x7fb0200ff3c0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0201a0100 con 0x7fb0200ff3c0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.363+0000 7fb0167fc640 1 -- 172.21.15.79:0/2380220640 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb01c018810 con 0x7fb0200ff3c0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.364+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0201a0460 con 0x7fb0200ff3c0 2024-06-01T00:54:29.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.365+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0201008a0 con 0x7fb0200ff3c0 2024-06-01T00:54:29.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.365+0000 7fb0167fc640 1 -- 172.21.15.79:0/2380220640 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb01c018a80 con 0x7fb0200ff3c0 2024-06-01T00:54:29.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.366+0000 7fb0167fc640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb00c06aef0 0x7fb00c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:29.368 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.366+0000 7fb024adc640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb00c06aef0 0x7fb00c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:29.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.366+0000 7fb0167fc640 1 -- 172.21.15.79:0/2380220640 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb01c089e50 con 0x7fb0200ff3c0 2024-06-01T00:54:29.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.366+0000 7fb024adc640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb00c06aef0 0x7fb00c06d3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb010000c00 tx=0x7fb010005e60 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:29.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.370+0000 7fb0167fc640 1 -- 172.21.15.79:0/2380220640 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb01c055430 con 0x7fb0200ff3c0 2024-06-01T00:54:29.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.579+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb02004e210 con 0x7fb0200ff3c0 2024-06-01T00:54:29.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.580+0000 7fb0167fc640 1 -- 172.21.15.79:0/2380220640 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb01c058a50 con 0x7fb0200ff3c0 2024-06-01T00:54:29.582 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:29.582 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:29.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb00c06aef0 msgr2=0x7fb00c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:29.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb00c06aef0 0x7fb00c06d3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb010000c00 tx=0x7fb010005e60 comp rx=0 tx=0).stop 2024-06-01T00:54:29.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ff3c0 msgr2=0x7fb020197090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:29.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ff3c0 0x7fb020197090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00c8a0 tx=0x7fb01c00cd60 comp rx=0 tx=0).stop 2024-06-01T00:54:29.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 shutdown_connections 2024-06-01T00:54:29.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb00c06aef0 0x7fb00c06d3b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:29.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb0200ffd70 0x7fb0201975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:29.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.581+0000 7fb027541640 1 --2- 172.21.15.79:0/2380220640 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0200ff3c0 0x7fb020197090 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:29.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.582+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 >> 172.21.15.79:0/2380220640 conn(0x7fb0200fac90 msgr2=0x7fb0200fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:29.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.582+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 shutdown_connections 2024-06-01T00:54:29.584 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:29.582+0000 7fb027541640 1 -- 172.21.15.79:0/2380220640 wait complete. 2024-06-01T00:54:30.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:30 smithi160 ceph-mon[36639]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:30.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:30 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2380220640' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:30.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:30 smithi079 ceph-mon[29356]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:30.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:30 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2380220640' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:32 smithi160 ceph-mon[36639]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:32.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:32 smithi079 ceph-mon[29356]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:33.192 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:33.408 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:34.779 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:34 smithi079 ceph-mon[29356]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:34.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:34 smithi160 ceph-mon[36639]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:35.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.085+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/4117630658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 msgr2=0x7f7b74103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:35.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.085+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/4117630658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b74103da0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009880 tx=0x7f7b6002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.085+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/4117630658 shutdown_connections 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.085+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/4117630658 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b74103da0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.085+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/4117630658 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b740ff3a0 0x7f7b740ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.085+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/4117630658 >> 172.21.15.79:0/4117630658 conn(0x7f7b740fac50 msgr2=0x7f7b740fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.085+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/4117630658 shutdown_connections 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.086+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/4117630658 wait complete. 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.086+0000 7f7b7ab1b640 1 Processor -- start 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.086+0000 7f7b7ab1b640 1 -- start start 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b7ab1b640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b740ff3a0 0x7f7b74196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b7ab1b640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b741968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b7ab1b640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b74196f40 con 0x7f7b740ffd50 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b7ab1b640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7419ac60 con 0x7f7b740ff3a0 2024-06-01T00:54:35.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b73fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b741968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b73fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b741968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37228/0 (socket says 172.21.15.79:37228) 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b73fff640 1 -- 172.21.15.79:0/3664848628 learned_addr learned my addr 172.21.15.79:0/3664848628 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b788b7640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b740ff3a0 0x7f7b74196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b73fff640 1 -- 172.21.15.79:0/3664848628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b740ff3a0 msgr2=0x7f7b74196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b73fff640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b740ff3a0 0x7f7b74196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b73fff640 1 -- 172.21.15.79:0/3664848628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b60009560 con 0x7f7b740ffd50 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b788b7640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b740ff3a0 0x7f7b74196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b73fff640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b741968b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7b6002a9c0 tx=0x7f7b600055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b71ffb640 1 -- 172.21.15.79:0/3664848628 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b60004aa0 con 0x7f7b740ffd50 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b71ffb640 1 -- 172.21.15.79:0/3664848628 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b60004c00 con 0x7f7b740ffd50 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7419aee0 con 0x7f7b740ffd50 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.087+0000 7f7b71ffb640 1 -- 172.21.15.79:0/3664848628 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b60003a50 con 0x7f7b740ffd50 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.088+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7419b3d0 con 0x7f7b740ffd50 2024-06-01T00:54:35.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.089+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b74100880 con 0x7f7b740ffd50 2024-06-01T00:54:35.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.089+0000 7f7b71ffb640 1 -- 172.21.15.79:0/3664848628 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7b60031c90 con 0x7f7b740ffd50 2024-06-01T00:54:35.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.089+0000 7f7b71ffb640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b6406aef0 0x7f7b6406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:35.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.090+0000 7f7b71ffb640 1 -- 172.21.15.79:0/3664848628 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7b600aae00 con 0x7f7b740ffd50 2024-06-01T00:54:35.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.090+0000 7f7b788b7640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b6406aef0 0x7f7b6406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:35.093 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.090+0000 7f7b788b7640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b6406aef0 0x7f7b6406d3b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006fd0 tx=0x7f7b68008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:35.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.094+0000 7f7b71ffb640 1 -- 172.21.15.79:0/3664848628 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7b600763e0 con 0x7f7b740ffd50 2024-06-01T00:54:35.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.302+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b7404e210 con 0x7f7b740ffd50 2024-06-01T00:54:35.303 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.302+0000 7f7b71ffb640 1 -- 172.21.15.79:0/3664848628 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7b60079a00 con 0x7f7b740ffd50 2024-06-01T00:54:35.303 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:35.303 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:35.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b6406aef0 msgr2=0x7f7b6406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:35.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b6406aef0 0x7f7b6406d3b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006fd0 tx=0x7f7b68008040 comp rx=0 tx=0).stop 2024-06-01T00:54:35.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 msgr2=0x7f7b741968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:35.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b741968b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7b6002a9c0 tx=0x7f7b600055a0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 shutdown_connections 2024-06-01T00:54:35.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f7b6406aef0 0x7f7b6406d3b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7b740ffd50 0x7f7b741968b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 --2- 172.21.15.79:0/3664848628 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7b740ff3a0 0x7f7b74196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:35.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 >> 172.21.15.79:0/3664848628 conn(0x7f7b740fac50 msgr2=0x7f7b74068ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:35.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 shutdown_connections 2024-06-01T00:54:35.306 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:35.304+0000 7f7b7ab1b640 1 -- 172.21.15.79:0/3664848628 wait complete. 2024-06-01T00:54:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:35 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3664848628' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:35.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:35 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3664848628' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:35.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:36 smithi160 ceph-mon[36639]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:36.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:36 smithi079 ceph-mon[29356]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:38 smithi160 ceph-mon[36639]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:38.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:38 smithi079 ceph-mon[29356]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:39.049 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:39.267 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:40.734 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:40 smithi079 ceph-mon[29356]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:40.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:40 smithi160 ceph-mon[36639]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:40.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 -- 172.21.15.79:0/3085936034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100e10 msgr2=0x7f8c30104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:40.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 --2- 172.21.15.79:0/3085936034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100e10 0x7f8c30104e60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009880 tx=0x7f8c1802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:54:40.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 -- 172.21.15.79:0/3085936034 shutdown_connections 2024-06-01T00:54:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 --2- 172.21.15.79:0/3085936034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100e10 0x7f8c30104e60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 --2- 172.21.15.79:0/3085936034 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c30100460 0x7f8c30100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 -- 172.21.15.79:0/3085936034 >> 172.21.15.79:0/3085936034 conn(0x7f8c300fbd10 msgr2=0x7f8c300fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 -- 172.21.15.79:0/3085936034 shutdown_connections 2024-06-01T00:54:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.950+0000 7f8c3579d640 1 -- 172.21.15.79:0/3085936034 wait complete. 2024-06-01T00:54:40.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c3579d640 1 Processor -- start 2024-06-01T00:54:40.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c3579d640 1 -- start start 2024-06-01T00:54:40.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c3579d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100460 0x7f8c30196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:40.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c3579d640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c30100e10 0x7f8c30196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:40.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c3579d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30196d80 con 0x7f8c30100460 2024-06-01T00:54:40.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c3579d640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30196ef0 con 0x7f8c30100e10 2024-06-01T00:54:40.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c2effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100460 0x7f8c30196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:40.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c2effd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100460 0x7f8c30196240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37242/0 (socket says 172.21.15.79:37242) 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c2effd640 1 -- 172.21.15.79:0/4149313087 learned_addr learned my addr 172.21.15.79:0/4149313087 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c2effd640 1 -- 172.21.15.79:0/4149313087 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c30100e10 msgr2=0x7f8c30196780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c2e7fc640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c30100e10 0x7f8c30196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c2effd640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c30100e10 0x7f8c30196780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.951+0000 7f8c2effd640 1 -- 172.21.15.79:0/4149313087 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18009560 con 0x7f8c30100460 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.952+0000 7f8c2effd640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100460 0x7f8c30196240 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800cc60 tx=0x7f8c2800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.952+0000 7f8c27fff640 1 -- 172.21.15.79:0/4149313087 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c2800f450 con 0x7f8c30100460 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.952+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3019ac80 con 0x7f8c30100460 2024-06-01T00:54:40.956 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.952+0000 7f8c27fff640 1 -- 172.21.15.79:0/4149313087 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c2800fa70 con 0x7f8c30100460 2024-06-01T00:54:40.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.952+0000 7f8c27fff640 1 -- 172.21.15.79:0/4149313087 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c28018b80 con 0x7f8c30100460 2024-06-01T00:54:40.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.952+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3019b1d0 con 0x7f8c30100460 2024-06-01T00:54:40.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.953+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c30101940 con 0x7f8c30100460 2024-06-01T00:54:40.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.953+0000 7f8c27fff640 1 -- 172.21.15.79:0/4149313087 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f8c28018d00 con 0x7f8c30100460 2024-06-01T00:54:40.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.954+0000 7f8c27fff640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8c1c06aef0 0x7f8c1c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:40.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.954+0000 7f8c27fff640 1 -- 172.21.15.79:0/4149313087 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8c2808a150 con 0x7f8c30100460 2024-06-01T00:54:40.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.954+0000 7f8c2e7fc640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8c1c06aef0 0x7f8c1c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:40.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.954+0000 7f8c2e7fc640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8c1c06aef0 0x7f8c1c06d3b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c18000c00 tx=0x7f8c18004120 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:40.959 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:40.958+0000 7f8c27fff640 1 -- 172.21.15.79:0/4149313087 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8c28055730 con 0x7f8c30100460 2024-06-01T00:54:41.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.168+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8c30197550 con 0x7f8c30100460 2024-06-01T00:54:41.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.168+0000 7f8c27fff640 1 -- 172.21.15.79:0/4149313087 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8c28058d50 con 0x7f8c30100460 2024-06-01T00:54:41.169 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:41.169 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:41.171 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.170+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8c1c06aef0 msgr2=0x7f8c1c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.170+0000 7f8c3579d640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8c1c06aef0 0x7f8c1c06d3b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8c18000c00 tx=0x7f8c18004120 comp rx=0 tx=0).stop 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.170+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100460 msgr2=0x7f8c30196240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.170+0000 7f8c3579d640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100460 0x7f8c30196240 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800cc60 tx=0x7f8c2800a620 comp rx=0 tx=0).stop 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.171+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 shutdown_connections 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.171+0000 7f8c3579d640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8c1c06aef0 0x7f8c1c06d3b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.171+0000 7f8c3579d640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8c30100e10 0x7f8c30196780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.171+0000 7f8c3579d640 1 --2- 172.21.15.79:0/4149313087 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8c30100460 0x7f8c30196240 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.171+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 >> 172.21.15.79:0/4149313087 conn(0x7f8c300fbd10 msgr2=0x7f8c30103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.171+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 shutdown_connections 2024-06-01T00:54:41.172 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:41.171+0000 7f8c3579d640 1 -- 172.21.15.79:0/4149313087 wait complete. 2024-06-01T00:54:41.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:41 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4149313087' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:41.832 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:41 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4149313087' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:42.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:42 smithi160 ceph-mon[36639]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:42.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:42 smithi079 ceph-mon[29356]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:44 smithi160 ceph-mon[36639]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:44.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:44 smithi079 ceph-mon[29356]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:44.856 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:45.074 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:46.709 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:46 smithi079 ceph-mon[29356]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 -- 172.21.15.79:0/4084889041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ff380 msgr2=0x7f22680ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 --2- 172.21.15.79:0/4084889041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ff380 0x7f22680ff760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2264009c20 tx=0x7f226402a970 comp rx=0 tx=0).stop 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 -- 172.21.15.79:0/4084889041 shutdown_connections 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 --2- 172.21.15.79:0/4084889041 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22680ffd30 0x7f2268103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 --2- 172.21.15.79:0/4084889041 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ff380 0x7f22680ff760 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 -- 172.21.15.79:0/4084889041 >> 172.21.15.79:0/4084889041 conn(0x7f22680fac30 msgr2=0x7f22680fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 -- 172.21.15.79:0/4084889041 shutdown_connections 2024-06-01T00:54:46.713 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.712+0000 7f226fb4a640 1 -- 172.21.15.79:0/4084889041 wait complete. 2024-06-01T00:54:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226fb4a640 1 Processor -- start 2024-06-01T00:54:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226fb4a640 1 -- start start 2024-06-01T00:54:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226fb4a640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22680ff380 0x7f2268197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:46.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226fb4a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ffd30 0x7f2268197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226fb4a640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2268197b90 con 0x7f22680ffd30 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226fb4a640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2268197d00 con 0x7f22680ff380 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226d0e5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ffd30 0x7f2268197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226d0e5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ffd30 0x7f2268197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44842/0 (socket says 172.21.15.79:44842) 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226d0e5640 1 -- 172.21.15.79:0/1956191645 learned_addr learned my addr 172.21.15.79:0/1956191645 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.713+0000 7f226d8e6640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22680ff380 0x7f2268197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f226d0e5640 1 -- 172.21.15.79:0/1956191645 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22680ff380 msgr2=0x7f2268197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f226d0e5640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22680ff380 0x7f2268197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f226d0e5640 1 -- 172.21.15.79:0/1956191645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264009900 con 0x7f22680ffd30 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f226d0e5640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ffd30 0x7f2268197650 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f225800c8f0 tx=0x7f225800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f225effd640 1 -- 172.21.15.79:0/1956191645 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2258007b80 con 0x7f22680ffd30 2024-06-01T00:54:46.716 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22681a0180 con 0x7f22680ffd30 2024-06-01T00:54:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f225effd640 1 -- 172.21.15.79:0/1956191645 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2258007ce0 con 0x7f22680ffd30 2024-06-01T00:54:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f225effd640 1 -- 172.21.15.79:0/1956191645 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22580187b0 con 0x7f22680ffd30 2024-06-01T00:54:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.714+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22681a0590 con 0x7f22680ffd30 2024-06-01T00:54:46.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.715+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2268100860 con 0x7f22680ffd30 2024-06-01T00:54:46.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.715+0000 7f225effd640 1 -- 172.21.15.79:0/1956191645 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f225800f450 con 0x7f22680ffd30 2024-06-01T00:54:46.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.716+0000 7f225effd640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f225406aef0 0x7f225406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:46.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.716+0000 7f225effd640 1 -- 172.21.15.79:0/1956191645 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2258089cc0 con 0x7f22680ffd30 2024-06-01T00:54:46.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.716+0000 7f226d8e6640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f225406aef0 0x7f225406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:46.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.716+0000 7f226d8e6640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f225406aef0 0x7f225406d3b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2264000f80 tx=0x7f226403d040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:46.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.720+0000 7f225effd640 1 -- 172.21.15.79:0/1956191645 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f22580552a0 con 0x7f22680ffd30 2024-06-01T00:54:46.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:46 smithi160 ceph-mon[36639]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:46.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.927+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f226804e210 con 0x7f22680ffd30 2024-06-01T00:54:46.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.928+0000 7f225effd640 1 -- 172.21.15.79:0/1956191645 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f22580588c0 con 0x7f22680ffd30 2024-06-01T00:54:46.928 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:46.929 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:46.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f225406aef0 msgr2=0x7f225406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f225406aef0 0x7f225406d3b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2264000f80 tx=0x7f226403d040 comp rx=0 tx=0).stop 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ffd30 msgr2=0x7f2268197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ffd30 0x7f2268197650 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f225800c8f0 tx=0x7f225800cdb0 comp rx=0 tx=0).stop 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 shutdown_connections 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f225406aef0 0x7f225406d3b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f22680ffd30 0x7f2268197650 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 --2- 172.21.15.79:0/1956191645 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f22680ff380 0x7f2268197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 >> 172.21.15.79:0/1956191645 conn(0x7f22680fac30 msgr2=0x7f22680fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 shutdown_connections 2024-06-01T00:54:46.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:46.930+0000 7f226fb4a640 1 -- 172.21.15.79:0/1956191645 wait complete. 2024-06-01T00:54:47.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:47 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1956191645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:47.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:47 smithi160 ceph-mon[36639]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:47.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:47 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1956191645' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:47.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:47 smithi079 ceph-mon[29356]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:50.761 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:50.789 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:50 smithi079 ceph-mon[29356]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:50.789 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:50 smithi160 ceph-mon[36639]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:54:50.979 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:52.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 -- 172.21.15.79:0/3125490052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 msgr2=0x7f88b4100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:52.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/3125490052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b4100890 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009880 tx=0x7f88a402a4a0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 -- 172.21.15.79:0/3125490052 shutdown_connections 2024-06-01T00:54:52.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/3125490052 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b4100e60 0x7f88b4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/3125490052 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b4100890 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 -- 172.21.15.79:0/3125490052 >> 172.21.15.79:0/3125490052 conn(0x7f88b40fbd20 msgr2=0x7f88b40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:52.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 -- 172.21.15.79:0/3125490052 shutdown_connections 2024-06-01T00:54:52.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 -- 172.21.15.79:0/3125490052 wait complete. 2024-06-01T00:54:52.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 Processor -- start 2024-06-01T00:54:52.640 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.638+0000 7f88ba2c5640 1 -- start start 2024-06-01T00:54:52.641 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88ba2c5640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b41984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88ba2c5640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b4100e60 0x7f88b4198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88ba2c5640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b41990b0 con 0x7f88b41004b0 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88ba2c5640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b419cdd0 con 0x7f88b4100e60 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b41984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b41984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54442/0 (socket says 172.21.15.79:54442) 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b3fff640 1 -- 172.21.15.79:0/960295883 learned_addr learned my addr 172.21.15.79:0/960295883 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b37fe640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b4100e60 0x7f88b4198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b3fff640 1 -- 172.21.15.79:0/960295883 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b4100e60 msgr2=0x7f88b4198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b3fff640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b4100e60 0x7f88b4198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b3fff640 1 -- 172.21.15.79:0/960295883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4009560 con 0x7f88b41004b0 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.639+0000 7f88b3fff640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b41984e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f88a402a9c0 tx=0x7f88a40055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.640+0000 7f88b17fa640 1 -- 172.21.15.79:0/960295883 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88a4004aa0 con 0x7f88b41004b0 2024-06-01T00:54:52.642 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.640+0000 7f88b17fa640 1 -- 172.21.15.79:0/960295883 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88a4004c00 con 0x7f88b41004b0 2024-06-01T00:54:52.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.640+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b419d050 con 0x7f88b41004b0 2024-06-01T00:54:52.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.640+0000 7f88b17fa640 1 -- 172.21.15.79:0/960295883 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88a4003980 con 0x7f88b41004b0 2024-06-01T00:54:52.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.640+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b419d540 con 0x7f88b41004b0 2024-06-01T00:54:52.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.641+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b4101990 con 0x7f88b41004b0 2024-06-01T00:54:52.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.641+0000 7f88b17fa640 1 -- 172.21.15.79:0/960295883 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f88a4003b20 con 0x7f88b41004b0 2024-06-01T00:54:52.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.641+0000 7f88b17fa640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f889c06aef0 0x7f889c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:52.643 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.642+0000 7f88b17fa640 1 -- 172.21.15.79:0/960295883 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f88a40aaaf0 con 0x7f88b41004b0 2024-06-01T00:54:52.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.642+0000 7f88b37fe640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f889c06aef0 0x7f889c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:52.644 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.642+0000 7f88b37fe640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f889c06aef0 0x7f889c06d3b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88b4199a40 tx=0x7f88a800b390 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:52.646 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.646+0000 7f88b17fa640 1 -- 172.21.15.79:0/960295883 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f88a4076200 con 0x7f88b41004b0 2024-06-01T00:54:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:52 smithi160 ceph-mon[36639]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:52.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:52 smithi079 ceph-mon[29356]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.855+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f88b404e210 con 0x7f88b41004b0 2024-06-01T00:54:52.856 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.855+0000 7f88b17fa640 1 -- 172.21.15.79:0/960295883 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f88a4079820 con 0x7f88b41004b0 2024-06-01T00:54:52.856 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:52.856 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:52.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f889c06aef0 msgr2=0x7f889c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:52.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f889c06aef0 0x7f889c06d3b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f88b4199a40 tx=0x7f88a800b390 comp rx=0 tx=0).stop 2024-06-01T00:54:52.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 msgr2=0x7f88b41984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:52.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b41984e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f88a402a9c0 tx=0x7f88a40055a0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 shutdown_connections 2024-06-01T00:54:52.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f889c06aef0 0x7f889c06d3b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f88b4100e60 0x7f88b4198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 --2- 172.21.15.79:0/960295883 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88b41004b0 0x7f88b41984e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:52.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 >> 172.21.15.79:0/960295883 conn(0x7f88b40fbd20 msgr2=0x7f88b40fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:52.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 shutdown_connections 2024-06-01T00:54:52.859 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:52.857+0000 7f88ba2c5640 1 -- 172.21.15.79:0/960295883 wait complete. 2024-06-01T00:54:53.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:53 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/960295883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:53.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:53 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/960295883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:54.606 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:54 smithi079 ceph-mon[29356]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:54.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:54 smithi160 ceph-mon[36639]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:56.484 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:54:56.701 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:54:56.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:56 smithi160 ceph-mon[36639]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:56.823 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:56 smithi079 ceph-mon[29356]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:58.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 -- 172.21.15.79:0/2550833774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 msgr2=0x7fc4781017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:58.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2550833774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781017a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc468009c20 tx=0x7fc46802aa00 comp rx=0 tx=0).stop 2024-06-01T00:54:58.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 -- 172.21.15.79:0/2550833774 shutdown_connections 2024-06-01T00:54:58.328 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2550833774 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc478101ce0 0x7fc47810a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2550833774 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781017a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 -- 172.21.15.79:0/2550833774 >> 172.21.15.79:0/2550833774 conn(0x7fc478075e10 msgr2=0x7fc4780fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 -- 172.21.15.79:0/2550833774 shutdown_connections 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 -- 172.21.15.79:0/2550833774 wait complete. 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 Processor -- start 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.315+0000 7fc47d63e640 1 -- start start 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc47d63e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc47d63e640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc478101ce0 0x7fc478196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc47d63e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc478196eb0 con 0x7fc4781013c0 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc47d63e640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47819abd0 con 0x7fc478101ce0 2024-06-01T00:54:58.329 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc476ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc476ffd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54460/0 (socket says 172.21.15.79:54460) 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc476ffd640 1 -- 172.21.15.79:0/2342406579 learned_addr learned my addr 172.21.15.79:0/2342406579 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc4767fc640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc478101ce0 0x7fc478196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc476ffd640 1 -- 172.21.15.79:0/2342406579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc478101ce0 msgr2=0x7fc478196820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc476ffd640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc478101ce0 0x7fc478196820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc476ffd640 1 -- 172.21.15.79:0/2342406579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468009900 con 0x7fc4781013c0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc4767fc640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc478101ce0 0x7fc478196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc476ffd640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781962e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc46802af20 tx=0x7fc46802f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc45ffff640 1 -- 172.21.15.79:0/2342406579 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc46803e070 con 0x7fc4781013c0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47819ae50 con 0x7fc4781013c0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc45ffff640 1 -- 172.21.15.79:0/2342406579 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc46802fcd0 con 0x7fc4781013c0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc45ffff640 1 -- 172.21.15.79:0/2342406579 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc468043470 con 0x7fc4781013c0 2024-06-01T00:54:58.330 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.316+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47819b340 con 0x7fc4781013c0 2024-06-01T00:54:58.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.318+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc478068510 con 0x7fc4781013c0 2024-06-01T00:54:58.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.318+0000 7fc45ffff640 1 -- 172.21.15.79:0/2342406579 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fc46800a040 con 0x7fc4781013c0 2024-06-01T00:54:58.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.318+0000 7fc45ffff640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc46006aef0 0x7fc46006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:54:58.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.318+0000 7fc45ffff640 1 -- 172.21.15.79:0/2342406579 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc4680aaf50 con 0x7fc4781013c0 2024-06-01T00:54:58.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.318+0000 7fc4767fc640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc46006aef0 0x7fc46006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:54:58.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.319+0000 7fc4767fc640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc46006aef0 0x7fc46006d3b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc46c006fd0 tx=0x7fc46c008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:54:58.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.323+0000 7fc45ffff640 1 -- 172.21.15.79:0/2342406579 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc4680de900 con 0x7fc4781013c0 2024-06-01T00:54:58.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.531+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc47804e210 con 0x7fc4781013c0 2024-06-01T00:54:58.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.532+0000 7fc45ffff640 1 -- 172.21.15.79:0/2342406579 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc468076530 con 0x7fc4781013c0 2024-06-01T00:54:58.533 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:54:58.533 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:54:58.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc46006aef0 msgr2=0x7fc46006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:58.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc46006aef0 0x7fc46006d3b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc46c006fd0 tx=0x7fc46c008040 comp rx=0 tx=0).stop 2024-06-01T00:54:58.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 msgr2=0x7fc4781962e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781962e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc46802af20 tx=0x7fc46802f890 comp rx=0 tx=0).stop 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 shutdown_connections 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc46006aef0 0x7fc46006d3b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc478101ce0 0x7fc478196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 --2- 172.21.15.79:0/2342406579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc4781013c0 0x7fc4781962e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 >> 172.21.15.79:0/2342406579 conn(0x7fc478075e10 msgr2=0x7fc4781026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 shutdown_connections 2024-06-01T00:54:58.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:54:58.534+0000 7fc47d63e640 1 -- 172.21.15.79:0/2342406579 wait complete. 2024-06-01T00:54:58.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:58 smithi160 ceph-mon[36639]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:58.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:58 smithi079 ceph-mon[29356]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:54:59.564 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:54:59 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2342406579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:54:59.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:54:59 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2342406579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:00.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:00 smithi079 ceph-mon[29356]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:00.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:00 smithi160 ceph-mon[36639]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:02.128 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:02.349 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:02.717 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:02 smithi079 ceph-mon[29356]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:02.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:02 smithi160 ceph-mon[36639]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:03.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 -- 172.21.15.79:0/4103985141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ffd50 msgr2=0x7feaa0103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:03.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 --2- 172.21.15.79:0/4103985141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ffd50 0x7feaa0103da0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009880 tx=0x7fea8c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:55:03.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 -- 172.21.15.79:0/4103985141 shutdown_connections 2024-06-01T00:55:03.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 --2- 172.21.15.79:0/4103985141 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ffd50 0x7feaa0103da0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 --2- 172.21.15.79:0/4103985141 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaa00ff3a0 0x7feaa00ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 -- 172.21.15.79:0/4103985141 >> 172.21.15.79:0/4103985141 conn(0x7feaa00fac50 msgr2=0x7feaa00fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 -- 172.21.15.79:0/4103985141 shutdown_connections 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 -- 172.21.15.79:0/4103985141 wait complete. 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 Processor -- start 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.928+0000 7feaa720f640 1 -- start start 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa720f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ff3a0 0x7feaa0196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa720f640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaa00ffd50 0x7feaa01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa720f640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0196f60 con 0x7feaa00ff3a0 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa720f640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa019ac80 con 0x7feaa00ffd50 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa4fab640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ff3a0 0x7feaa0196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:03.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa4fab640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ff3a0 0x7feaa0196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37742/0 (socket says 172.21.15.79:37742) 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa4fab640 1 -- 172.21.15.79:0/3563209485 learned_addr learned my addr 172.21.15.79:0/3563209485 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7fea9ffff640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaa00ffd50 0x7feaa01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa4fab640 1 -- 172.21.15.79:0/3563209485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaa00ffd50 msgr2=0x7feaa01968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa4fab640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaa00ffd50 0x7feaa01968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa4fab640 1 -- 172.21.15.79:0/3563209485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c009560 con 0x7feaa00ff3a0 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa4fab640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ff3a0 0x7feaa0196390 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fea9400b420 tx=0x7fea9400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7fea9dffb640 1 -- 172.21.15.79:0/3563209485 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea94011800 con 0x7feaa00ff3a0 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7fea9dffb640 1 -- 172.21.15.79:0/3563209485 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea94011e20 con 0x7feaa00ff3a0 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa019af60 con 0x7feaa00ff3a0 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.929+0000 7fea9dffb640 1 -- 172.21.15.79:0/3563209485 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea9400f2e0 con 0x7feaa00ff3a0 2024-06-01T00:55:03.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.930+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa019b480 con 0x7feaa00ff3a0 2024-06-01T00:55:03.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.931+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa0100880 con 0x7feaa00ff3a0 2024-06-01T00:55:03.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.931+0000 7fea9dffb640 1 -- 172.21.15.79:0/3563209485 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fea94010380 con 0x7feaa00ff3a0 2024-06-01T00:55:03.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.931+0000 7fea9dffb640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fea9006aef0 0x7fea9006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:03.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.932+0000 7fea9dffb640 1 -- 172.21.15.79:0/3563209485 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fea94089570 con 0x7feaa00ff3a0 2024-06-01T00:55:03.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.932+0000 7fea9ffff640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fea9006aef0 0x7fea9006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:03.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.933+0000 7fea9ffff640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fea9006aef0 0x7fea9006d3b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009230 tx=0x7fea8c005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:03.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:03.936+0000 7fea9dffb640 1 -- 172.21.15.79:0/3563209485 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fea94054b50 con 0x7feaa00ff3a0 2024-06-01T00:55:04.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.144+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feaa004e210 con 0x7feaa00ff3a0 2024-06-01T00:55:04.145 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.144+0000 7fea9dffb640 1 -- 172.21.15.79:0/3563209485 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fea94058170 con 0x7feaa00ff3a0 2024-06-01T00:55:04.145 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:04.145 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:04.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.146+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fea9006aef0 msgr2=0x7fea9006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:04.147 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.146+0000 7feaa720f640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fea9006aef0 0x7fea9006d3b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009230 tx=0x7fea8c005a20 comp rx=0 tx=0).stop 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ff3a0 msgr2=0x7feaa0196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ff3a0 0x7feaa0196390 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fea9400b420 tx=0x7fea9400b8e0 comp rx=0 tx=0).stop 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 shutdown_connections 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fea9006aef0 0x7fea9006d3b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7feaa00ffd50 0x7feaa01968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 --2- 172.21.15.79:0/3563209485 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feaa00ff3a0 0x7feaa0196390 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 >> 172.21.15.79:0/3563209485 conn(0x7feaa00fac50 msgr2=0x7feaa0068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 shutdown_connections 2024-06-01T00:55:04.148 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:04.147+0000 7feaa720f640 1 -- 172.21.15.79:0/3563209485 wait complete. 2024-06-01T00:55:04.719 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:04 smithi079 ceph-mon[29356]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:04.719 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:04 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3563209485' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:04.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:04 smithi160 ceph-mon[36639]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:04.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:04 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3563209485' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:05.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:05.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:06.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:06 smithi160 ceph-mon[36639]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:06.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:06 smithi079 ceph-mon[29356]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:07.741 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:07.764 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:55:07.764 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:55:07.765 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:55:07.765 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:55:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:55:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:55:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:55:07.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:55:07.944 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:08.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:08 smithi160 ceph-mon[36639]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:08.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:08 smithi079 ceph-mon[29356]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:09.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/3468798240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 msgr2=0x7ff3ec10a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:09.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/3468798240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec10a220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8009880 tx=0x7ff3d802a530 comp rx=0 tx=0).stop 2024-06-01T00:55:09.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/3468798240 shutdown_connections 2024-06-01T00:55:09.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/3468798240 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec10a220 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:09.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/3468798240 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3ec101410 0x7ff3ec1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:09.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/3468798240 >> 172.21.15.79:0/3468798240 conn(0x7ff3ec0fac80 msgr2=0x7ff3ec0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:09.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/3468798240 shutdown_connections 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.572+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/3468798240 wait complete. 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f2ee1640 1 Processor -- start 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f2ee1640 1 -- start start 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f2ee1640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3ec101410 0x7ff3ec196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f2ee1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f2ee1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec196f00 con 0x7ff3ec101d30 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f2ee1640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec19ac20 con 0x7ff3ec101410 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3ebfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3ebfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37760/0 (socket says 172.21.15.79:37760) 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3ebfff640 1 -- 172.21.15.79:0/973036989 learned_addr learned my addr 172.21.15.79:0/973036989 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f0c7d640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3ec101410 0x7ff3ec196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3ebfff640 1 -- 172.21.15.79:0/973036989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3ec101410 msgr2=0x7ff3ec196330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:09.581 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3ebfff640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3ec101410 0x7ff3ec196330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3ebfff640 1 -- 172.21.15.79:0/973036989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d8009560 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.573+0000 7ff3f0c7d640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3ec101410 0x7ff3ec196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.574+0000 7ff3ebfff640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec196870 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff3d802aa50 tx=0x7ff3d80055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.574+0000 7ff3e9ffb640 1 -- 172.21.15.79:0/973036989 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d8004aa0 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.574+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec19aea0 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.574+0000 7ff3e9ffb640 1 -- 172.21.15.79:0/973036989 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3d8004c00 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.574+0000 7ff3e9ffb640 1 -- 172.21.15.79:0/973036989 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3d80039e0 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.574+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec19b390 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.575+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec069060 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.575+0000 7ff3e9ffb640 1 -- 172.21.15.79:0/973036989 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7ff3d802ad50 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.576+0000 7ff3e9ffb640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff3dc06aef0 0x7ff3dc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.576+0000 7ff3e9ffb640 1 -- 172.21.15.79:0/973036989 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff3d80aaaa0 con 0x7ff3ec101d30 2024-06-01T00:55:09.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.576+0000 7ff3f0c7d640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff3dc06aef0 0x7ff3dc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:09.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.576+0000 7ff3f0c7d640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff3dc06aef0 0x7ff3dc06d3b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0006fd0 tx=0x7ff3e0008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:09.583 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.581+0000 7ff3e9ffb640 1 -- 172.21.15.79:0/973036989 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7ff3d8075fe0 con 0x7ff3ec101d30 2024-06-01T00:55:09.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.788+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff3ec04e210 con 0x7ff3ec101d30 2024-06-01T00:55:09.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.788+0000 7ff3e9ffb640 1 -- 172.21.15.79:0/973036989 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff3d8079600 con 0x7ff3ec101d30 2024-06-01T00:55:09.790 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:09.790 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:09.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff3dc06aef0 msgr2=0x7ff3dc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:09.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff3dc06aef0 0x7ff3dc06d3b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0006fd0 tx=0x7ff3e0008040 comp rx=0 tx=0).stop 2024-06-01T00:55:09.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 msgr2=0x7ff3ec196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:09.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec196870 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff3d802aa50 tx=0x7ff3d80055a0 comp rx=0 tx=0).stop 2024-06-01T00:55:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 shutdown_connections 2024-06-01T00:55:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7ff3dc06aef0 0x7ff3dc06d3b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff3ec101d30 0x7ff3ec196870 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 --2- 172.21.15.79:0/973036989 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7ff3ec101410 0x7ff3ec196330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.790+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 >> 172.21.15.79:0/973036989 conn(0x7ff3ec0fac80 msgr2=0x7ff3ec102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.791+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 shutdown_connections 2024-06-01T00:55:09.793 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:09.791+0000 7ff3f2ee1640 1 -- 172.21.15.79:0/973036989 wait complete. 2024-06-01T00:55:10.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:10 smithi160 ceph-mon[36639]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:10.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:10 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/973036989' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:10.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:10 smithi079 ceph-mon[29356]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:10.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:10 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/973036989' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:12 smithi160 ceph-mon[36639]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:12.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:12 smithi079 ceph-mon[29356]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:13.411 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:13.629 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:14.794 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:14 smithi079 ceph-mon[29356]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:14.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:14 smithi160 ceph-mon[36639]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 -- 172.21.15.79:0/2452119060 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101410 msgr2=0x7fa6d41017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/2452119060 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101410 0x7fa6d41017f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009880 tx=0x7fa6c402a530 comp rx=0 tx=0).stop 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 -- 172.21.15.79:0/2452119060 shutdown_connections 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/2452119060 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa6d4101d30 0x7fa6d410a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/2452119060 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101410 0x7fa6d41017f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 -- 172.21.15.79:0/2452119060 >> 172.21.15.79:0/2452119060 conn(0x7fa6d40fac80 msgr2=0x7fa6d40fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 -- 172.21.15.79:0/2452119060 shutdown_connections 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 -- 172.21.15.79:0/2452119060 wait complete. 2024-06-01T00:55:15.209 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.193+0000 7fa6d99f0640 1 Processor -- start 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d99f0640 1 -- start start 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d99f0640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa6d4101410 0x7fa6d4196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d99f0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101d30 0x7fa6d4196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d99f0640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d4196ee0 con 0x7fa6d4101d30 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d99f0640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d419ac00 con 0x7fa6d4101410 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d27fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101d30 0x7fa6d4196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d27fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101d30 0x7fa6d4196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34478/0 (socket says 172.21.15.79:34478) 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d27fc640 1 -- 172.21.15.79:0/268577300 learned_addr learned my addr 172.21.15.79:0/268577300 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d2ffd640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa6d4101410 0x7fa6d4196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d27fc640 1 -- 172.21.15.79:0/268577300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa6d4101410 msgr2=0x7fa6d4196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d27fc640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa6d4101410 0x7fa6d4196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.194+0000 7fa6d27fc640 1 -- 172.21.15.79:0/268577300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4009560 con 0x7fa6d4101d30 2024-06-01T00:55:15.210 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.195+0000 7fa6d27fc640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101d30 0x7fa6d4196850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800b340 tx=0x7fa6c800b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.195+0000 7fa6d89ee640 1 -- 172.21.15.79:0/268577300 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6c8011800 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.195+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d419aee0 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.195+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d419b430 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.195+0000 7fa6d89ee640 1 -- 172.21.15.79:0/268577300 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6c8011e20 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.195+0000 7fa6d89ee640 1 -- 172.21.15.79:0/268577300 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6c800f2e0 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.196+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d4069060 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.196+0000 7fa6d89ee640 1 -- 172.21.15.79:0/268577300 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fa6c8010380 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.197+0000 7fa6d89ee640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa6bc06aef0 0x7fa6bc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.197+0000 7fa6d2ffd640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa6bc06aef0 0x7fa6bc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.197+0000 7fa6d89ee640 1 -- 172.21.15.79:0/268577300 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa6c8089890 con 0x7fa6d4101d30 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.198+0000 7fa6d2ffd640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa6bc06aef0 0x7fa6bc06d3b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000c00 tx=0x7fa6c4004120 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:15.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.201+0000 7fa6d89ee640 1 -- 172.21.15.79:0/268577300 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fa6c8054e70 con 0x7fa6d4101d30 2024-06-01T00:55:15.418 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.417+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa6d404e210 con 0x7fa6d4101d30 2024-06-01T00:55:15.419 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.418+0000 7fa6d89ee640 1 -- 172.21.15.79:0/268577300 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa6c8058490 con 0x7fa6d4101d30 2024-06-01T00:55:15.419 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:15.419 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa6bc06aef0 msgr2=0x7fa6bc06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa6bc06aef0 0x7fa6bc06d3b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000c00 tx=0x7fa6c4004120 comp rx=0 tx=0).stop 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101d30 msgr2=0x7fa6d4196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101d30 0x7fa6d4196850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800b340 tx=0x7fa6c800b800 comp rx=0 tx=0).stop 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 shutdown_connections 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fa6bc06aef0 0x7fa6bc06d3b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa6d4101d30 0x7fa6d4196850 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 --2- 172.21.15.79:0/268577300 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fa6d4101410 0x7fa6d4196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 >> 172.21.15.79:0/268577300 conn(0x7fa6d40fac80 msgr2=0x7fa6d4102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 shutdown_connections 2024-06-01T00:55:15.421 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:15.420+0000 7fa6d99f0640 1 -- 172.21.15.79:0/268577300 wait complete. 2024-06-01T00:55:15.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:15 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/268577300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:15.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:15 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/268577300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:16.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:16 smithi160 ceph-mon[36639]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:16.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:16 smithi079 ceph-mon[29356]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:18 smithi160 ceph-mon[36639]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:18.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:18 smithi079 ceph-mon[29356]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:19.135 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:19.352 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:20.726 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:20 smithi079 ceph-mon[29356]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:20.726 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:20 smithi160 ceph-mon[36639]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:20.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.932+0000 7f9722d52640 1 -- 172.21.15.79:0/1608239819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ff360 msgr2=0x7f971c0ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:20.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.932+0000 7f9722d52640 1 --2- 172.21.15.79:0/1608239819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ff360 0x7f971c0ff740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9714009880 tx=0x7f971402a4a0 comp rx=0 tx=0).stop 2024-06-01T00:55:20.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 -- 172.21.15.79:0/1608239819 shutdown_connections 2024-06-01T00:55:20.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 --2- 172.21.15.79:0/1608239819 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f971c0ffd10 0x7f971c103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 --2- 172.21.15.79:0/1608239819 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ff360 0x7f971c0ff740 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 -- 172.21.15.79:0/1608239819 >> 172.21.15.79:0/1608239819 conn(0x7f971c075e10 msgr2=0x7f971c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 -- 172.21.15.79:0/1608239819 shutdown_connections 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 -- 172.21.15.79:0/1608239819 wait complete. 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 Processor -- start 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 -- start start 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f971c0ff360 0x7f971c196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ffd10 0x7f971c196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c196f10 con 0x7f971c0ffd10 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f9722d52640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c19ac30 con 0x7f971c0ff360 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.933+0000 7f971bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ffd10 0x7f971c196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f971bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ffd10 0x7f971c196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34490/0 (socket says 172.21.15.79:34490) 2024-06-01T00:55:20.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f971bfff640 1 -- 172.21.15.79:0/672462312 learned_addr learned my addr 172.21.15.79:0/672462312 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f9720aee640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f971c0ff360 0x7f971c196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f971bfff640 1 -- 172.21.15.79:0/672462312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f971c0ff360 msgr2=0x7f971c196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f971bfff640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f971c0ff360 0x7f971c196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f971bfff640 1 -- 172.21.15.79:0/672462312 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714009560 con 0x7f971c0ffd10 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f9720aee640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f971c0ff360 0x7f971c196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f971bfff640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ffd10 0x7f971c196880 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f970400b3d0 tx=0x7f970400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f9719ffb640 1 -- 172.21.15.79:0/672462312 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9704011800 con 0x7f971c0ffd10 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f9719ffb640 1 -- 172.21.15.79:0/672462312 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9704011e20 con 0x7f971c0ffd10 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c19af10 con 0x7f971c0ffd10 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f9719ffb640 1 -- 172.21.15.79:0/672462312 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f970400f2e0 con 0x7f971c0ffd10 2024-06-01T00:55:20.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.934+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c19b460 con 0x7f971c0ffd10 2024-06-01T00:55:20.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.935+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971c100840 con 0x7f971c0ffd10 2024-06-01T00:55:20.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.936+0000 7f9719ffb640 1 -- 172.21.15.79:0/672462312 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f9704010380 con 0x7f971c0ffd10 2024-06-01T00:55:20.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.937+0000 7f9719ffb640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f970806aef0 0x7f970806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:20.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.937+0000 7f9720aee640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f970806aef0 0x7f970806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:20.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.937+0000 7f9719ffb640 1 -- 172.21.15.79:0/672462312 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9704089930 con 0x7f971c0ffd10 2024-06-01T00:55:20.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.937+0000 7f9720aee640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f970806aef0 0x7f970806d3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9714000c00 tx=0x7f9714005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:20.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:20.941+0000 7f9719ffb640 1 -- 172.21.15.79:0/672462312 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f9704054f10 con 0x7f971c0ffd10 2024-06-01T00:55:21.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.149+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f971c04e210 con 0x7f971c0ffd10 2024-06-01T00:55:21.150 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.149+0000 7f9719ffb640 1 -- 172.21.15.79:0/672462312 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9704058530 con 0x7f971c0ffd10 2024-06-01T00:55:21.150 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:21.150 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:21.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f970806aef0 msgr2=0x7f970806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:21.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f970806aef0 0x7f970806d3b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9714000c00 tx=0x7f9714005a20 comp rx=0 tx=0).stop 2024-06-01T00:55:21.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ffd10 msgr2=0x7f971c196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:21.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ffd10 0x7f971c196880 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f970400b3d0 tx=0x7f970400b890 comp rx=0 tx=0).stop 2024-06-01T00:55:21.152 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 shutdown_connections 2024-06-01T00:55:21.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f970806aef0 0x7f970806d3b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:21.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f971c0ffd10 0x7f971c196880 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:21.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 --2- 172.21.15.79:0/672462312 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f971c0ff360 0x7f971c196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:21.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 >> 172.21.15.79:0/672462312 conn(0x7f971c075e10 msgr2=0x7f971c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:21.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 shutdown_connections 2024-06-01T00:55:21.153 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:21.151+0000 7f9722d52640 1 -- 172.21.15.79:0/672462312 wait complete. 2024-06-01T00:55:21.758 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:21 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/672462312' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:21.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:21 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/672462312' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:22.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:22 smithi160 ceph-mon[36639]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:22.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:22 smithi079 ceph-mon[29356]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:24.783 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:24 smithi160 ceph-mon[36639]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:24.810 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:24 smithi079 ceph-mon[29356]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:24.996 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:26.677 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:26 smithi079 ceph-mon[29356]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:26.679 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.678+0000 7f8dac24d640 1 -- 172.21.15.79:0/872066242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 msgr2=0x7f8da41017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.678+0000 7f8dac24d640 1 --2- 172.21.15.79:0/872066242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da41017e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009880 tx=0x7f8da002a530 comp rx=0 tx=0).stop 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 -- 172.21.15.79:0/872066242 shutdown_connections 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 --2- 172.21.15.79:0/872066242 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8da4101d20 0x7f8da410a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 --2- 172.21.15.79:0/872066242 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da41017e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 -- 172.21.15.79:0/872066242 >> 172.21.15.79:0/872066242 conn(0x7f8da40fac50 msgr2=0x7f8da40fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 -- 172.21.15.79:0/872066242 shutdown_connections 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 -- 172.21.15.79:0/872066242 wait complete. 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 Processor -- start 2024-06-01T00:55:26.680 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 -- start start 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da40ff060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8da4101d20 0x7f8da40ff5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da40ffae0 con 0x7f8da4101400 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.679+0000 7f8dac24d640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da40ffc50 con 0x7f8da4101d20 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da9fe9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da40ff060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da9fe9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da40ff060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54098/0 (socket says 172.21.15.79:54098) 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da9fe9640 1 -- 172.21.15.79:0/4212018418 learned_addr learned my addr 172.21.15.79:0/4212018418 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da97e8640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8da4101d20 0x7f8da40ff5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:26.682 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da9fe9640 1 -- 172.21.15.79:0/4212018418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8da4101d20 msgr2=0x7f8da40ff5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da9fe9640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8da4101d20 0x7f8da40ff5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da9fe9640 1 -- 172.21.15.79:0/4212018418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0009560 con 0x7f8da4101400 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8da9fe9640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da40ff060 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8da002aa50 tx=0x7f8da00055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8d9affd640 1 -- 172.21.15.79:0/4212018418 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da0004aa0 con 0x7f8da4101400 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da40ffed0 con 0x7f8da4101400 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8d9affd640 1 -- 172.21.15.79:0/4212018418 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8da0004c00 con 0x7f8da4101400 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8d9affd640 1 -- 172.21.15.79:0/4212018418 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8da0003980 con 0x7f8da4101400 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.680+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da41a2600 con 0x7f8da4101400 2024-06-01T00:55:26.683 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.681+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da4069080 con 0x7f8da4101400 2024-06-01T00:55:26.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.682+0000 7f8d9affd640 1 -- 172.21.15.79:0/4212018418 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f8da002ad50 con 0x7f8da4101400 2024-06-01T00:55:26.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.682+0000 7f8d9affd640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8d9006aef0 0x7f8d9006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:26.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.682+0000 7f8da97e8640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8d9006aef0 0x7f8d9006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:26.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.682+0000 7f8d9affd640 1 -- 172.21.15.79:0/4212018418 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8da00aa7d0 con 0x7f8da4101400 2024-06-01T00:55:26.684 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.683+0000 7f8da97e8640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8d9006aef0 0x7f8d9006d3b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8da4100790 tx=0x7f8d9400b2e0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:26.687 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.687+0000 7f8d9affd640 1 -- 172.21.15.79:0/4212018418 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f8da0075e60 con 0x7f8da4101400 2024-06-01T00:55:26.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:26 smithi160 ceph-mon[36639]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:26.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.894+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8da4100540 con 0x7f8da4101400 2024-06-01T00:55:26.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.894+0000 7f8d9affd640 1 -- 172.21.15.79:0/4212018418 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8da0079480 con 0x7f8da4101400 2024-06-01T00:55:26.895 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:26.895 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:26.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8d9006aef0 msgr2=0x7f8d9006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:26.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8d9006aef0 0x7f8d9006d3b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8da4100790 tx=0x7f8d9400b2e0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 msgr2=0x7f8da40ff060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:26.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da40ff060 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8da002aa50 tx=0x7f8da00055a0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 shutdown_connections 2024-06-01T00:55:26.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f8d9006aef0 0x7f8d9006d3b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f8da4101d20 0x7f8da40ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 --2- 172.21.15.79:0/4212018418 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8da4101400 0x7f8da40ff060 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:26.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 >> 172.21.15.79:0/4212018418 conn(0x7f8da40fac50 msgr2=0x7f8da41026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:26.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 shutdown_connections 2024-06-01T00:55:26.898 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:26.896+0000 7f8dac24d640 1 -- 172.21.15.79:0/4212018418 wait complete. 2024-06-01T00:55:27.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:27 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4212018418' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:27.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:27 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4212018418' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:28.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:28 smithi160 ceph-mon[36639]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:28.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:28 smithi079 ceph-mon[29356]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:30.554 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:30.769 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:30.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:30 smithi160 ceph-mon[36639]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:30.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:30 smithi079 ceph-mon[29356]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 -- 172.21.15.79:0/300664000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 msgr2=0x7f0c640ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 --2- 172.21.15.79:0/300664000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c640ff7a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009880 tx=0x7f0c5c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 -- 172.21.15.79:0/300664000 shutdown_connections 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 --2- 172.21.15.79:0/300664000 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c640ffd70 0x7f0c64103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 --2- 172.21.15.79:0/300664000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c640ff7a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 -- 172.21.15.79:0/300664000 >> 172.21.15.79:0/300664000 conn(0x7f0c640facb0 msgr2=0x7f0c640fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 -- 172.21.15.79:0/300664000 shutdown_connections 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.309+0000 7f0c6b356640 1 -- 172.21.15.79:0/300664000 wait complete. 2024-06-01T00:55:32.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c6b356640 1 Processor -- start 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c6b356640 1 -- start start 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c6b356640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c64195100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c6b356640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c640ffd70 0x7f0c64195640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c6b356640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c64195cd0 con 0x7f0c640ff3c0 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c6b356640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c641999f0 con 0x7f0c640ffd70 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c690f2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c64195100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c690f2640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c64195100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58570/0 (socket says 172.21.15.79:58570) 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c690f2640 1 -- 172.21.15.79:0/1296560373 learned_addr learned my addr 172.21.15.79:0/1296560373 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.310+0000 7f0c688f1640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c640ffd70 0x7f0c64195640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c690f2640 1 -- 172.21.15.79:0/1296560373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c640ffd70 msgr2=0x7f0c64195640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c690f2640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c640ffd70 0x7f0c64195640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c690f2640 1 -- 172.21.15.79:0/1296560373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c009560 con 0x7f0c640ff3c0 2024-06-01T00:55:32.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c690f2640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c64195100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c02a9c0 tx=0x7f0c5c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:32.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c627fc640 1 -- 172.21.15.79:0/1296560373 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c5c004aa0 con 0x7f0c640ff3c0 2024-06-01T00:55:32.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c627fc640 1 -- 172.21.15.79:0/1296560373 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c5c004c00 con 0x7f0c640ff3c0 2024-06-01T00:55:32.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c64199c70 con 0x7f0c640ff3c0 2024-06-01T00:55:32.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c627fc640 1 -- 172.21.15.79:0/1296560373 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c5c003980 con 0x7f0c640ff3c0 2024-06-01T00:55:32.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.311+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6419a160 con 0x7f0c640ff3c0 2024-06-01T00:55:32.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.312+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c641008a0 con 0x7f0c640ff3c0 2024-06-01T00:55:32.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.312+0000 7f0c627fc640 1 -- 172.21.15.79:0/1296560373 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f0c5c003b20 con 0x7f0c640ff3c0 2024-06-01T00:55:32.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.313+0000 7f0c627fc640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0c4c06aef0 0x7f0c4c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:32.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.313+0000 7f0c688f1640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0c4c06aef0 0x7f0c4c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:32.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.313+0000 7f0c627fc640 1 -- 172.21.15.79:0/1296560373 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0c5c0aaaf0 con 0x7f0c640ff3c0 2024-06-01T00:55:32.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.314+0000 7f0c688f1640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0c4c06aef0 0x7f0c4c06d3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0c64196660 tx=0x7f0c5800b390 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:32.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.317+0000 7f0c627fc640 1 -- 172.21.15.79:0/1296560373 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f0c5c076200 con 0x7f0c640ff3c0 2024-06-01T00:55:32.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.528+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0c6404e210 con 0x7f0c640ff3c0 2024-06-01T00:55:32.529 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.528+0000 7f0c627fc640 1 -- 172.21.15.79:0/1296560373 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0c6404e210 con 0x7f0c640ff3c0 2024-06-01T00:55:32.529 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:32.529 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:32.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0c4c06aef0 msgr2=0x7f0c4c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:32.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0c4c06aef0 0x7f0c4c06d3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0c64196660 tx=0x7f0c5800b390 comp rx=0 tx=0).stop 2024-06-01T00:55:32.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 msgr2=0x7f0c64195100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:32.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c64195100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c02a9c0 tx=0x7f0c5c0055a0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.531 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 shutdown_connections 2024-06-01T00:55:32.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f0c4c06aef0 0x7f0c4c06d3b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f0c640ffd70 0x7f0c64195640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 --2- 172.21.15.79:0/1296560373 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c640ff3c0 0x7f0c64195100 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:32.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 >> 172.21.15.79:0/1296560373 conn(0x7f0c640facb0 msgr2=0x7f0c640fbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:32.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 shutdown_connections 2024-06-01T00:55:32.532 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:32.530+0000 7f0c6b356640 1 -- 172.21.15.79:0/1296560373 wait complete. 2024-06-01T00:55:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:32 smithi160 ceph-mon[36639]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:32.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:32 smithi079 ceph-mon[29356]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:33.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:33 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1296560373' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:33.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:33 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1296560373' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:34.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:34 smithi079 ceph-mon[29356]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:34.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:34 smithi160 ceph-mon[36639]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:35.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:36.157 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:36.374 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:36.754 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:36 smithi079 ceph-mon[29356]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:36 smithi160 ceph-mon[36639]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:37.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 -- 172.21.15.79:0/3302400065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 msgr2=0x7fb8740ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:37.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/3302400065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb8740ff780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009c20 tx=0x7fb86c02a970 comp rx=0 tx=0).stop 2024-06-01T00:55:37.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 -- 172.21.15.79:0/3302400065 shutdown_connections 2024-06-01T00:55:37.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/3302400065 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8740ffd50 0x7fb874103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:37.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/3302400065 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb8740ff780 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 -- 172.21.15.79:0/3302400065 >> 172.21.15.79:0/3302400065 conn(0x7fb8740fac50 msgr2=0x7fb8740fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 -- 172.21.15.79:0/3302400065 shutdown_connections 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.857+0000 7fb87b4a1640 1 -- 172.21.15.79:0/3302400065 wait complete. 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87b4a1640 1 Processor -- start 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87b4a1640 1 -- start start 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87b4a1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb874196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87b4a1640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8740ffd50 0x7fb874196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87b4a1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb874196ed0 con 0x7fb8740ff3a0 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87b4a1640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87419abf0 con 0x7fb8740ffd50 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87923d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb874196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:37.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87923d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb874196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58584/0 (socket says 172.21.15.79:58584) 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87923d640 1 -- 172.21.15.79:0/1711853374 learned_addr learned my addr 172.21.15.79:0/1711853374 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87923d640 1 -- 172.21.15.79:0/1711853374 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8740ffd50 msgr2=0x7fb874196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87923d640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8740ffd50 0x7fb874196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87923d640 1 -- 172.21.15.79:0/1711853374 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c009900 con 0x7fb8740ff3a0 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.858+0000 7fb87923d640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb874196300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb86c02ae90 tx=0x7fb86c0308a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.859+0000 7fb86a7fc640 1 -- 172.21.15.79:0/1711853374 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86c02f590 con 0x7fb8740ff3a0 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.859+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87419ae70 con 0x7fb8740ff3a0 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.859+0000 7fb86a7fc640 1 -- 172.21.15.79:0/1711853374 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb86c030ca0 con 0x7fb8740ff3a0 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.859+0000 7fb86a7fc640 1 -- 172.21.15.79:0/1711853374 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86c039680 con 0x7fb8740ff3a0 2024-06-01T00:55:37.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.859+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87419b360 con 0x7fb8740ff3a0 2024-06-01T00:55:37.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.860+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb874100880 con 0x7fb8740ff3a0 2024-06-01T00:55:37.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.860+0000 7fb86a7fc640 1 -- 172.21.15.79:0/1711853374 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb86c00a040 con 0x7fb8740ff3a0 2024-06-01T00:55:37.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.860+0000 7fb86a7fc640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb86006aef0 0x7fb86006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:37.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.861+0000 7fb878a3c640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb86006aef0 0x7fb86006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:37.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.861+0000 7fb86a7fc640 1 -- 172.21.15.79:0/1711853374 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb86c02c040 con 0x7fb8740ff3a0 2024-06-01T00:55:37.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.861+0000 7fb878a3c640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb86006aef0 0x7fb86006d3b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb874197860 tx=0x7fb864006c90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:37.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:37.865+0000 7fb86a7fc640 1 -- 172.21.15.79:0/1711853374 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb86c076640 con 0x7fb8740ff3a0 2024-06-01T00:55:38.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.072+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb87404e210 con 0x7fb8740ff3a0 2024-06-01T00:55:38.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.072+0000 7fb86a7fc640 1 -- 172.21.15.79:0/1711853374 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb86c079c60 con 0x7fb8740ff3a0 2024-06-01T00:55:38.073 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:38.073 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb86006aef0 msgr2=0x7fb86006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb86006aef0 0x7fb86006d3b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb874197860 tx=0x7fb864006c90 comp rx=0 tx=0).stop 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 msgr2=0x7fb874196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb874196300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb86c02ae90 tx=0x7fb86c0308a0 comp rx=0 tx=0).stop 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 shutdown_connections 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb86006aef0 0x7fb86006d3b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8740ffd50 0x7fb874196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 --2- 172.21.15.79:0/1711853374 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8740ff3a0 0x7fb874196300 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 >> 172.21.15.79:0/1711853374 conn(0x7fb8740fac50 msgr2=0x7fb874109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 shutdown_connections 2024-06-01T00:55:38.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:38.074+0000 7fb87b4a1640 1 -- 172.21.15.79:0/1711853374 wait complete. 2024-06-01T00:55:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:38 smithi160 ceph-mon[36639]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:38 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1711853374' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:38.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:38 smithi079 ceph-mon[29356]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:38.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:38 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1711853374' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:40.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:40 smithi160 ceph-mon[36639]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:40.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:40 smithi079 ceph-mon[29356]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:41.688 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:41.904 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:42.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:42 smithi079 ceph-mon[29356]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:42.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:42 smithi160 ceph-mon[36639]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:43.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 -- 172.21.15.79:0/2973749024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 msgr2=0x7fab94103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/2973749024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab94103da0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009880 tx=0x7fab7c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 -- 172.21.15.79:0/2973749024 shutdown_connections 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/2973749024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab94103da0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/2973749024 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab940ff3a0 0x7fab940ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 -- 172.21.15.79:0/2973749024 >> 172.21.15.79:0/2973749024 conn(0x7fab940fac50 msgr2=0x7fab940fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 -- 172.21.15.79:0/2973749024 shutdown_connections 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.381+0000 7fab9a3f1640 1 -- 172.21.15.79:0/2973749024 wait complete. 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab9a3f1640 1 Processor -- start 2024-06-01T00:55:43.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab9a3f1640 1 -- start start 2024-06-01T00:55:43.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab9a3f1640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab940ff3a0 0x7fab94195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:43.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab9a3f1640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab941957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab9a3f1640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94195e50 con 0x7fab940ffd50 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab9a3f1640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94199b70 con 0x7fab940ff3a0 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab937fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab941957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab937fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab941957c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37540/0 (socket says 172.21.15.79:37540) 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.382+0000 7fab937fe640 1 -- 172.21.15.79:0/3897410494 learned_addr learned my addr 172.21.15.79:0/3897410494 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab93fff640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab940ff3a0 0x7fab94195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab937fe640 1 -- 172.21.15.79:0/3897410494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab940ff3a0 msgr2=0x7fab94195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab937fe640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab940ff3a0 0x7fab94195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab937fe640 1 -- 172.21.15.79:0/3897410494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c009560 con 0x7fab940ffd50 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab93fff640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab940ff3a0 0x7fab94195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab937fe640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab941957c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fab7c02a9c0 tx=0x7fab7c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:43.386 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab917fa640 1 -- 172.21.15.79:0/3897410494 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab7c004aa0 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab917fa640 1 -- 172.21.15.79:0/3897410494 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab7c004c00 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab94199df0 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab917fa640 1 -- 172.21.15.79:0/3897410494 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab7c003a50 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.383+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9419a2e0 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.384+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab94100880 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.385+0000 7fab917fa640 1 -- 172.21.15.79:0/3897410494 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fab7c031c90 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.385+0000 7fab917fa640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab8006aef0 0x7fab8006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.385+0000 7fab917fa640 1 -- 172.21.15.79:0/3897410494 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fab7c0aad00 con 0x7fab940ffd50 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.385+0000 7fab93fff640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab8006aef0 0x7fab8006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:43.387 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.385+0000 7fab93fff640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab8006aef0 0x7fab8006d3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab84006fd0 tx=0x7fab84008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:43.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.389+0000 7fab917fa640 1 -- 172.21.15.79:0/3897410494 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fab7c0ae030 con 0x7fab940ffd50 2024-06-01T00:55:43.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.600+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fab941076b0 con 0x7fab940ffd50 2024-06-01T00:55:43.601 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.600+0000 7fab917fa640 1 -- 172.21.15.79:0/3897410494 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fab7c079df0 con 0x7fab940ffd50 2024-06-01T00:55:43.601 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:43.601 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:43.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab8006aef0 msgr2=0x7fab8006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:43.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab8006aef0 0x7fab8006d3b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fab84006fd0 tx=0x7fab84008040 comp rx=0 tx=0).stop 2024-06-01T00:55:43.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 msgr2=0x7fab941957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:43.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab941957c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fab7c02a9c0 tx=0x7fab7c0055a0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 shutdown_connections 2024-06-01T00:55:43.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fab8006aef0 0x7fab8006d3b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fab940ffd50 0x7fab941957c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 --2- 172.21.15.79:0/3897410494 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fab940ff3a0 0x7fab94195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:43.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.602+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 >> 172.21.15.79:0/3897410494 conn(0x7fab940fac50 msgr2=0x7fab940fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:43.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.603+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 shutdown_connections 2024-06-01T00:55:43.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:43.603+0000 7fab9a3f1640 1 -- 172.21.15.79:0/3897410494 wait complete. 2024-06-01T00:55:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:44 smithi160 ceph-mon[36639]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:44 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3897410494' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:44.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:44 smithi079 ceph-mon[29356]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:44.852 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:44 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3897410494' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:46.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:46 smithi160 ceph-mon[36639]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:46.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:46 smithi079 ceph-mon[29356]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:47.251 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:47.460 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:48.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:48 smithi160 ceph-mon[36639]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:48.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:48 smithi079 ceph-mon[29356]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:49.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.068+0000 7f05c2d27640 1 -- 172.21.15.79:0/462963788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100470 msgr2=0x7f05bc100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:49.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.068+0000 7f05c2d27640 1 --2- 172.21.15.79:0/462963788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100470 0x7f05bc100850 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f05ac009c20 tx=0x7f05ac02a970 comp rx=0 tx=0).stop 2024-06-01T00:55:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.068+0000 7f05c2d27640 1 -- 172.21.15.79:0/462963788 shutdown_connections 2024-06-01T00:55:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.068+0000 7f05c2d27640 1 --2- 172.21.15.79:0/462963788 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05bc100e20 0x7f05bc104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.068+0000 7f05c2d27640 1 --2- 172.21.15.79:0/462963788 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100470 0x7f05bc100850 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.068+0000 7f05c2d27640 1 -- 172.21.15.79:0/462963788 >> 172.21.15.79:0/462963788 conn(0x7f05bc0fbd20 msgr2=0x7f05bc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 -- 172.21.15.79:0/462963788 shutdown_connections 2024-06-01T00:55:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 -- 172.21.15.79:0/462963788 wait complete. 2024-06-01T00:55:49.074 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 Processor -- start 2024-06-01T00:55:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 -- start start 2024-06-01T00:55:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05bc100470 0x7f05bc196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100e20 0x7f05bc196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc196e20 con 0x7f05bc100e20 2024-06-01T00:55:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05c2d27640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc19aaf0 con 0x7f05bc100470 2024-06-01T00:55:49.075 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05bbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100e20 0x7f05bc196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05bbfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100e20 0x7f05bc196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:37554/0 (socket says 172.21.15.79:37554) 2024-06-01T00:55:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.069+0000 7f05bbfff640 1 -- 172.21.15.79:0/4095542955 learned_addr learned my addr 172.21.15.79:0/4095542955 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05c0ac3640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05bc100470 0x7f05bc196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05bbfff640 1 -- 172.21.15.79:0/4095542955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05bc100470 msgr2=0x7f05bc196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05bbfff640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05bc100470 0x7f05bc196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:49.076 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05bbfff640 1 -- 172.21.15.79:0/4095542955 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac009900 con 0x7f05bc100e20 2024-06-01T00:55:49.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05bbfff640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100e20 0x7f05bc196790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f05b000d6c0 tx=0x7f05b000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:49.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05b9ffb640 1 -- 172.21.15.79:0/4095542955 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05b00051f0 con 0x7f05bc100e20 2024-06-01T00:55:49.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05b9ffb640 1 -- 172.21.15.79:0/4095542955 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05b0009ce0 con 0x7f05bc100e20 2024-06-01T00:55:49.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc19add0 con 0x7f05bc100e20 2024-06-01T00:55:49.077 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05b9ffb640 1 -- 172.21.15.79:0/4095542955 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05b0010460 con 0x7f05bc100e20 2024-06-01T00:55:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.070+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc19b320 con 0x7f05bc100e20 2024-06-01T00:55:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.071+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc101950 con 0x7f05bc100e20 2024-06-01T00:55:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.072+0000 7f05b9ffb640 1 -- 172.21.15.79:0/4095542955 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f05b0005350 con 0x7f05bc100e20 2024-06-01T00:55:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.072+0000 7f05b9ffb640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f05a406aef0 0x7f05a406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.072+0000 7f05b9ffb640 1 -- 172.21.15.79:0/4095542955 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f05b00895f0 con 0x7f05bc100e20 2024-06-01T00:55:49.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.072+0000 7f05c0ac3640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f05a406aef0 0x7f05a406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:49.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.073+0000 7f05c0ac3640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f05a406aef0 0x7f05a406d3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f05ac005fa0 tx=0x7f05ac005f10 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:49.079 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.076+0000 7f05b9ffb640 1 -- 172.21.15.79:0/4095542955 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f05b0054c50 con 0x7f05bc100e20 2024-06-01T00:55:49.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.284+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f05bc04e210 con 0x7f05bc100e20 2024-06-01T00:55:49.285 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:49.285 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:49.285 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.284+0000 7f05b9ffb640 1 -- 172.21.15.79:0/4095542955 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f05b00200d0 con 0x7f05bc100e20 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f05a406aef0 msgr2=0x7f05a406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f05a406aef0 0x7f05a406d3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f05ac005fa0 tx=0x7f05ac005f10 comp rx=0 tx=0).stop 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100e20 msgr2=0x7f05bc196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100e20 0x7f05bc196790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f05b000d6c0 tx=0x7f05b000db80 comp rx=0 tx=0).stop 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 shutdown_connections 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f05a406aef0 0x7f05a406d3b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f05bc100e20 0x7f05bc196790 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 --2- 172.21.15.79:0/4095542955 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f05bc100470 0x7f05bc196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:49.287 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 >> 172.21.15.79:0/4095542955 conn(0x7f05bc0fbd20 msgr2=0x7f05bc0fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:49.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 shutdown_connections 2024-06-01T00:55:49.288 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:49.286+0000 7f05c2d27640 1 -- 172.21.15.79:0/4095542955 wait complete. 2024-06-01T00:55:49.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:49 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4095542955' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:49.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:49 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4095542955' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:50 smithi160 ceph-mon[36639]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:50.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:50 smithi079 ceph-mon[29356]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:50.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:55:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:52 smithi160 ceph-mon[36639]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:52.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:52 smithi079 ceph-mon[29356]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:52.909 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:53.125 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:54.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 -- 172.21.15.79:0/93379897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1062f0 msgr2=0x7fe8bc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:54.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/93379897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1062f0 0x7fe8bc1066d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009880 tx=0x7fe8ac02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 -- 172.21.15.79:0/93379897 shutdown_connections 2024-06-01T00:55:54.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/93379897 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc0fe9f0 0x7fe8bc0fee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/93379897 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1062f0 0x7fe8bc1066d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 -- 172.21.15.79:0/93379897 >> 172.21.15.79:0/93379897 conn(0x7fe8bc074430 msgr2=0x7fe8bc074840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:54.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 -- 172.21.15.79:0/93379897 shutdown_connections 2024-06-01T00:55:54.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.691+0000 7fe8c1d81640 1 -- 172.21.15.79:0/93379897 wait complete. 2024-06-01T00:55:54.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8c1d81640 1 Processor -- start 2024-06-01T00:55:54.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8c1d81640 1 -- start start 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8c1d81640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fe9f0 0x7fe8bc1961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8c1d81640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc1062f0 0x7fe8bc1966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8c1d81640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc196d70 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8c1d81640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc19aa90 con 0x7fe8bc1062f0 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fe9f0 0x7fe8bc1961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fe9f0 0x7fe8bc1961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33512/0 (socket says 172.21.15.79:33512) 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8bb7fe640 1 -- 172.21.15.79:0/1295299419 learned_addr learned my addr 172.21.15.79:0/1295299419 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:55:54.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.692+0000 7fe8baffd640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc1062f0 0x7fe8bc1966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8bb7fe640 1 -- 172.21.15.79:0/1295299419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc1062f0 msgr2=0x7fe8bc1966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8bb7fe640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc1062f0 0x7fe8bc1966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8bb7fe640 1 -- 172.21.15.79:0/1295299419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ac009560 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8bb7fe640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fe9f0 0x7fe8bc1961a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac02a9c0 tx=0x7fe8ac0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8b8ff9640 1 -- 172.21.15.79:0/1295299419 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8ac004aa0 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8b8ff9640 1 -- 172.21.15.79:0/1295299419 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8ac004c00 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8b8ff9640 1 -- 172.21.15.79:0/1295299419 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8ac003ab0 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8bc19ad10 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.693+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc19b1b0 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.694+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc068410 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.694+0000 7fe8b8ff9640 1 -- 172.21.15.79:0/1295299419 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe8ac031d20 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.695+0000 7fe8b8ff9640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:55:54.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.695+0000 7fe8b8ff9640 1 -- 172.21.15.79:0/1295299419 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe8ac0aac60 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.695+0000 7fe8baffd640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:55:54.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.695+0000 7fe8baffd640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc197700 tx=0x7fe8b000a2d0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:55:54.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.699+0000 7fe8b8ff9640 1 -- 172.21.15.79:0/1295299419 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe8ac0762f0 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:54 smithi160 ceph-mon[36639]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:54.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:54 smithi079 ceph-mon[29356]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:54.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.909+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe8bc04e210 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.910+0000 7fe8b8ff9640 1 -- 172.21.15.79:0/1295299419 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe8ac079910 con 0x7fe8bc0fe9f0 2024-06-01T00:55:54.912 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:55:54.912 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:55:54.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.911+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 msgr2=0x7fe8a406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:54.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc197700 tx=0x7fe8b000a2d0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fe9f0 msgr2=0x7fe8bc1961a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:55:54.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fe9f0 0x7fe8bc1961a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac02a9c0 tx=0x7fe8ac0055a0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 shutdown_connections 2024-06-01T00:55:54.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc1062f0 0x7fe8bc1966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 --2- 172.21.15.79:0/1295299419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fe9f0 0x7fe8bc1961a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:55:54.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 >> 172.21.15.79:0/1295299419 conn(0x7fe8bc074430 msgr2=0x7fe8bc0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:55:54.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 shutdown_connections 2024-06-01T00:55:54.914 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:55:54.912+0000 7fe8c1d81640 1 -- 172.21.15.79:0/1295299419 wait complete. 2024-06-01T00:55:55.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:55 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1295299419' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:55.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:55 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1295299419' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:55:56.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:56 smithi160 ceph-mon[36639]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:56.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:56 smithi079 ceph-mon[29356]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:58.530 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:55:58.746 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:55:58.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:58 smithi160 ceph-mon[36639]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:58.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:58 smithi079 ceph-mon[29356]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:55:59.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:55:59 smithi079 ceph-mon[29356]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:00.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:55:59 smithi160 ceph-mon[36639]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 -- 172.21.15.79:0/2692840572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ff360 msgr2=0x7fc1280ff740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 --2- 172.21.15.79:0/2692840572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ff360 0x7fc1280ff740 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc118009880 tx=0x7fc11802a4a0 comp rx=0 tx=0).stop 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 -- 172.21.15.79:0/2692840572 shutdown_connections 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 --2- 172.21.15.79:0/2692840572 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1280ffd10 0x7fc128103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 --2- 172.21.15.79:0/2692840572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ff360 0x7fc1280ff740 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 -- 172.21.15.79:0/2692840572 >> 172.21.15.79:0/2692840572 conn(0x7fc128075e10 msgr2=0x7fc1280fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 -- 172.21.15.79:0/2692840572 shutdown_connections 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.329+0000 7fc12e36c640 1 -- 172.21.15.79:0/2692840572 wait complete. 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc12e36c640 1 Processor -- start 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc12e36c640 1 -- start start 2024-06-01T00:56:00.332 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc12e36c640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1280ff360 0x7fc1281962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc12e36c640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ffd10 0x7fc128196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc12e36c640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc128196ec0 con 0x7fc1280ffd10 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc12e36c640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12819abe0 con 0x7fc1280ff360 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc11ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ffd10 0x7fc128196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc11ffff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ffd10 0x7fc128196830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33538/0 (socket says 172.21.15.79:33538) 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc11ffff640 1 -- 172.21.15.79:0/1630158296 learned_addr learned my addr 172.21.15.79:0/1630158296 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc127fff640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1280ff360 0x7fc1281962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc11ffff640 1 -- 172.21.15.79:0/1630158296 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1280ff360 msgr2=0x7fc1281962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc11ffff640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1280ff360 0x7fc1281962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.330+0000 7fc11ffff640 1 -- 172.21.15.79:0/1630158296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc118009560 con 0x7fc1280ffd10 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.331+0000 7fc127fff640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1280ff360 0x7fc1281962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.331+0000 7fc11ffff640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ffd10 0x7fc128196830 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc11000b3d0 tx=0x7fc11000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.331+0000 7fc125ffb640 1 -- 172.21.15.79:0/1630158296 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc110011800 con 0x7fc1280ffd10 2024-06-01T00:56:00.333 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.331+0000 7fc125ffb640 1 -- 172.21.15.79:0/1630158296 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc110011e20 con 0x7fc1280ffd10 2024-06-01T00:56:00.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.331+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12819aec0 con 0x7fc1280ffd10 2024-06-01T00:56:00.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.331+0000 7fc125ffb640 1 -- 172.21.15.79:0/1630158296 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11000f2e0 con 0x7fc1280ffd10 2024-06-01T00:56:00.334 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.331+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12819b410 con 0x7fc1280ffd10 2024-06-01T00:56:00.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.332+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc128100840 con 0x7fc1280ffd10 2024-06-01T00:56:00.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.332+0000 7fc125ffb640 1 -- 172.21.15.79:0/1630158296 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fc110011960 con 0x7fc1280ffd10 2024-06-01T00:56:00.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.333+0000 7fc125ffb640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc11406aef0 0x7fc11406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:00.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.333+0000 7fc125ffb640 1 -- 172.21.15.79:0/1630158296 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc1100898e0 con 0x7fc1280ffd10 2024-06-01T00:56:00.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.333+0000 7fc127fff640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc11406aef0 0x7fc11406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:00.335 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.333+0000 7fc127fff640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc11406aef0 0x7fc11406d3b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:00.338 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.337+0000 7fc125ffb640 1 -- 172.21.15.79:0/1630158296 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fc110054ec0 con 0x7fc1280ffd10 2024-06-01T00:56:00.543 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.543+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc12804e210 con 0x7fc1280ffd10 2024-06-01T00:56:00.544 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.543+0000 7fc125ffb640 1 -- 172.21.15.79:0/1630158296 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc1100584e0 con 0x7fc1280ffd10 2024-06-01T00:56:00.544 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:00.544 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:00.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc11406aef0 msgr2=0x7fc11406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:00.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc11406aef0 0x7fc11406d3b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc118000c00 tx=0x7fc118005a20 comp rx=0 tx=0).stop 2024-06-01T00:56:00.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ffd10 msgr2=0x7fc128196830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:00.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ffd10 0x7fc128196830 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc11000b3d0 tx=0x7fc11000b890 comp rx=0 tx=0).stop 2024-06-01T00:56:00.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 shutdown_connections 2024-06-01T00:56:00.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fc11406aef0 0x7fc11406d3b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:00.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc1280ffd10 0x7fc128196830 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:00.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 --2- 172.21.15.79:0/1630158296 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fc1280ff360 0x7fc1281962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:00.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 >> 172.21.15.79:0/1630158296 conn(0x7fc128075e10 msgr2=0x7fc1280683f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:00.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 shutdown_connections 2024-06-01T00:56:00.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:00.545+0000 7fc12e36c640 1 -- 172.21.15.79:0/1630158296 wait complete. 2024-06-01T00:56:00.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:00 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1630158296' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:01.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:00 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1630158296' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:01.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:01 smithi079 ceph-mon[29356]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:02.054 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:01 smithi160 ceph-mon[36639]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:04.170 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:04.387 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:04.509 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:04 smithi079 ceph-mon[29356]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:04.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:04 smithi160 ceph-mon[36639]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:05.739 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:05 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:05.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:05 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 -- 172.21.15.79:0/2186911389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fea40 msgr2=0x7fe8bc0feec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 --2- 172.21.15.79:0/2186911389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fea40 0x7fe8bc0feec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009880 tx=0x7fe8b002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 -- 172.21.15.79:0/2186911389 shutdown_connections 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 --2- 172.21.15.79:0/2186911389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc0fea40 0x7fe8bc0feec0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 --2- 172.21.15.79:0/2186911389 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc1048c0 0x7fe8bc104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 -- 172.21.15.79:0/2186911389 >> 172.21.15.79:0/2186911389 conn(0x7fe8bc074460 msgr2=0x7fe8bc074870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 -- 172.21.15.79:0/2186911389 shutdown_connections 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 -- 172.21.15.79:0/2186911389 wait complete. 2024-06-01T00:56:06.081 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 Processor -- start 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.079+0000 7fe8c2863640 1 -- start start 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8c2863640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc0fea40 0x7fe8bc196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8c2863640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1048c0 0x7fe8bc196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8c2863640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc196ee0 con 0x7fe8bc1048c0 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8c2863640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8bc19ac00 con 0x7fe8bc0fea40 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1048c0 0x7fe8bc196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bb7fe640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1048c0 0x7fe8bc196850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48264/0 (socket says 172.21.15.79:48264) 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bb7fe640 1 -- 172.21.15.79:0/876378175 learned_addr learned my addr 172.21.15.79:0/876378175 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bbfff640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc0fea40 0x7fe8bc196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bb7fe640 1 -- 172.21.15.79:0/876378175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc0fea40 msgr2=0x7fe8bc196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:06.082 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bb7fe640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc0fea40 0x7fe8bc196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bb7fe640 1 -- 172.21.15.79:0/876378175 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b0009560 con 0x7fe8bc1048c0 2024-06-01T00:56:06.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bbfff640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc0fea40 0x7fe8bc196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-01T00:56:06.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.080+0000 7fe8bb7fe640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1048c0 0x7fe8bc196850 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe8b002a9c0 tx=0x7fe8b00055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:06.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.081+0000 7fe8b97fa640 1 -- 172.21.15.79:0/876378175 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8b0004aa0 con 0x7fe8bc1048c0 2024-06-01T00:56:06.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.081+0000 7fe8b97fa640 1 -- 172.21.15.79:0/876378175 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8b0004c00 con 0x7fe8bc1048c0 2024-06-01T00:56:06.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.081+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8bc19ae80 con 0x7fe8bc1048c0 2024-06-01T00:56:06.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.081+0000 7fe8b97fa640 1 -- 172.21.15.79:0/876378175 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8b0003ab0 con 0x7fe8bc1048c0 2024-06-01T00:56:06.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.081+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8bc19b370 con 0x7fe8bc1048c0 2024-06-01T00:56:06.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.082+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8bc100020 con 0x7fe8bc1048c0 2024-06-01T00:56:06.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.082+0000 7fe8b97fa640 1 -- 172.21.15.79:0/876378175 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fe8b0003ce0 con 0x7fe8bc1048c0 2024-06-01T00:56:06.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.082+0000 7fe8b97fa640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:06.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.083+0000 7fe8b97fa640 1 -- 172.21.15.79:0/876378175 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe8b00aad40 con 0x7fe8bc1048c0 2024-06-01T00:56:06.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.083+0000 7fe8bbfff640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:06.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.083+0000 7fe8bbfff640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009930 tx=0x7fe8ac008040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:06.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.087+0000 7fe8b97fa640 1 -- 172.21.15.79:0/876378175 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fe8b0076400 con 0x7fe8bc1048c0 2024-06-01T00:56:06.295 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.294+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe8bc04e210 con 0x7fe8bc1048c0 2024-06-01T00:56:06.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.295+0000 7fe8b97fa640 1 -- 172.21.15.79:0/876378175 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe8b0079a20 con 0x7fe8bc1048c0 2024-06-01T00:56:06.296 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:06.296 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:06.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 msgr2=0x7fe8a406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:06.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009930 tx=0x7fe8ac008040 comp rx=0 tx=0).stop 2024-06-01T00:56:06.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1048c0 msgr2=0x7fe8bc196850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:06.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1048c0 0x7fe8bc196850 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe8b002a9c0 tx=0x7fe8b00055a0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.298 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 shutdown_connections 2024-06-01T00:56:06.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fe8a406aef0 0x7fe8a406d3b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe8bc1048c0 0x7fe8bc196850 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 --2- 172.21.15.79:0/876378175 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fe8bc0fea40 0x7fe8bc196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:06.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 >> 172.21.15.79:0/876378175 conn(0x7fe8bc074460 msgr2=0x7fe8bc0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:06.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 shutdown_connections 2024-06-01T00:56:06.299 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:06.297+0000 7fe8c2863640 1 -- 172.21.15.79:0/876378175 wait complete. 2024-06-01T00:56:06.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:06 smithi079 ceph-mon[29356]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:06.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:06 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/876378175' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:06.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:06 smithi160 ceph-mon[36639]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:06.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:06 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/876378175' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:07.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:07 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:56:07.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:07 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T00:56:08.706 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:08 smithi160 ceph-mon[36639]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:08.706 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:08 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:56:08.706 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:08 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:56:08.706 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:08 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:56:08.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:08 smithi079 ceph-mon[29356]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:08.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:08 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T00:56:08.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:08 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T00:56:08.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:08 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:56:10.102 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:10.323 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:10.750 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:10 smithi079 ceph-mon[29356]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:10.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:10 smithi160 ceph-mon[36639]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:11.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.910+0000 7f70338a3640 1 -- 172.21.15.79:0/3071969708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ffd10 msgr2=0x7f702c103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:11.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.910+0000 7f70338a3640 1 --2- 172.21.15.79:0/3071969708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ffd10 0x7f702c103d60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f701c009880 tx=0x7f701c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:56:11.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.910+0000 7f70338a3640 1 -- 172.21.15.79:0/3071969708 shutdown_connections 2024-06-01T00:56:11.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.910+0000 7f70338a3640 1 --2- 172.21.15.79:0/3071969708 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ffd10 0x7f702c103d60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:11.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.910+0000 7f70338a3640 1 --2- 172.21.15.79:0/3071969708 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f702c0ff360 0x7f702c0ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:11.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.910+0000 7f70338a3640 1 -- 172.21.15.79:0/3071969708 >> 172.21.15.79:0/3071969708 conn(0x7f702c075e10 msgr2=0x7f702c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:11.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 -- 172.21.15.79:0/3071969708 shutdown_connections 2024-06-01T00:56:11.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 -- 172.21.15.79:0/3071969708 wait complete. 2024-06-01T00:56:11.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 Processor -- start 2024-06-01T00:56:11.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 -- start start 2024-06-01T00:56:11.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ff360 0x7f702c1962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:11.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f702c0ffd10 0x7f702c196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:11.917 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c196eb0 con 0x7f702c0ff360 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f70338a3640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c19abd0 con 0x7f702c0ffd10 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f703163f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ff360 0x7f702c1962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f703163f640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ff360 0x7f702c1962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48292/0 (socket says 172.21.15.79:48292) 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.911+0000 7f703163f640 1 -- 172.21.15.79:0/973003820 learned_addr learned my addr 172.21.15.79:0/973003820 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f7030e3e640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f702c0ffd10 0x7f702c196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f703163f640 1 -- 172.21.15.79:0/973003820 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f702c0ffd10 msgr2=0x7f702c196820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f703163f640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f702c0ffd10 0x7f702c196820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:11.918 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f703163f640 1 -- 172.21.15.79:0/973003820 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c009560 con 0x7f702c0ff360 2024-06-01T00:56:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f703163f640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ff360 0x7f702c1962e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f702800ba50 tx=0x7f702800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f70227fc640 1 -- 172.21.15.79:0/973003820 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f702800cb30 con 0x7f702c0ff360 2024-06-01T00:56:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f70227fc640 1 -- 172.21.15.79:0/973003820 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f702800cc90 con 0x7f702c0ff360 2024-06-01T00:56:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c19aeb0 con 0x7f702c0ff360 2024-06-01T00:56:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f70227fc640 1 -- 172.21.15.79:0/973003820 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7028007740 con 0x7f702c0ff360 2024-06-01T00:56:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.912+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702c19b3d0 con 0x7f702c0ff360 2024-06-01T00:56:11.919 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.913+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702c100840 con 0x7f702c0ff360 2024-06-01T00:56:11.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.913+0000 7f70227fc640 1 -- 172.21.15.79:0/973003820 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7028014b90 con 0x7f702c0ff360 2024-06-01T00:56:11.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.914+0000 7f70227fc640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f701806aef0 0x7f701806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:11.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.914+0000 7f7030e3e640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f701806aef0 0x7f701806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:11.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.914+0000 7f70227fc640 1 -- 172.21.15.79:0/973003820 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7028089520 con 0x7f702c0ff360 2024-06-01T00:56:11.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.914+0000 7f7030e3e640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f701806aef0 0x7f701806d3b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f701c009230 tx=0x7f701c005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:11.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:11.918+0000 7f70227fc640 1 -- 172.21.15.79:0/973003820 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f70280586c0 con 0x7f702c0ff360 2024-06-01T00:56:12.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.129+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f702c04e210 con 0x7f702c0ff360 2024-06-01T00:56:12.131 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.130+0000 7f70227fc640 1 -- 172.21.15.79:0/973003820 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7028058250 con 0x7f702c0ff360 2024-06-01T00:56:12.131 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:12.131 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:12.132 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f701806aef0 msgr2=0x7f701806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f701806aef0 0x7f701806d3b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f701c009230 tx=0x7f701c005a20 comp rx=0 tx=0).stop 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ff360 msgr2=0x7f702c1962e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ff360 0x7f702c1962e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f702800ba50 tx=0x7f702800bf10 comp rx=0 tx=0).stop 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 shutdown_connections 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f701806aef0 0x7f701806d3b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f702c0ffd10 0x7f702c196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 --2- 172.21.15.79:0/973003820 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f702c0ff360 0x7f702c1962e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 >> 172.21.15.79:0/973003820 conn(0x7f702c075e10 msgr2=0x7f702c067de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 shutdown_connections 2024-06-01T00:56:12.133 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:12.132+0000 7f70338a3640 1 -- 172.21.15.79:0/973003820 wait complete. 2024-06-01T00:56:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:12 smithi160 ceph-mon[36639]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:12.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:12 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/973003820' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:12.845 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:12 smithi079 ceph-mon[29356]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:12.846 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:12 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/973003820' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:14.806 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:14 smithi160 ceph-mon[36639]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:14.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:14 smithi079 ceph-mon[29356]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:15.867 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:16.085 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:16.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:16 smithi160 ceph-mon[36639]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:16.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:16 smithi079 ceph-mon[29356]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 -- 172.21.15.79:0/4114348967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101460 msgr2=0x7fca44101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 --2- 172.21.15.79:0/4114348967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101460 0x7fca44101840 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fca380098b0 tx=0x7fca3802a560 comp rx=0 tx=0).stop 2024-06-01T00:56:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 -- 172.21.15.79:0/4114348967 shutdown_connections 2024-06-01T00:56:17.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 --2- 172.21.15.79:0/4114348967 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fca44101d80 0x7fca4410a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 --2- 172.21.15.79:0/4114348967 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101460 0x7fca44101840 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 -- 172.21.15.79:0/4114348967 >> 172.21.15.79:0/4114348967 conn(0x7fca44075e10 msgr2=0x7fca440fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 -- 172.21.15.79:0/4114348967 shutdown_connections 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 -- 172.21.15.79:0/4114348967 wait complete. 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 Processor -- start 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.777+0000 7fca4ac29640 1 -- start start 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca4ac29640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fca44101460 0x7fca440ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca4ac29640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101d80 0x7fca440ff600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca4ac29640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca440ffb40 con 0x7fca44101d80 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca4ac29640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca440ffcb0 con 0x7fca44101460 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca43fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101d80 0x7fca440ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:17.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca43fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101d80 0x7fca440ff600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45616/0 (socket says 172.21.15.79:45616) 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca43fff640 1 -- 172.21.15.79:0/1167371856 learned_addr learned my addr 172.21.15.79:0/1167371856 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca489c5640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fca44101460 0x7fca440ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca43fff640 1 -- 172.21.15.79:0/1167371856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fca44101460 msgr2=0x7fca440ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca43fff640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fca44101460 0x7fca440ff0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca43fff640 1 -- 172.21.15.79:0/1167371856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38009560 con 0x7fca44101d80 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca43fff640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101d80 0x7fca440ff600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fca3000c8f0 tx=0x7fca3000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca41ffb640 1 -- 172.21.15.79:0/1167371856 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca30007b80 con 0x7fca44101d80 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca41ffb640 1 -- 172.21.15.79:0/1167371856 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca300054e0 con 0x7fca44101d80 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca441a2480 con 0x7fca44101d80 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.778+0000 7fca41ffb640 1 -- 172.21.15.79:0/1167371856 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3000f450 con 0x7fca44101d80 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.779+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca441a27e0 con 0x7fca44101d80 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.779+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca44068470 con 0x7fca44101d80 2024-06-01T00:56:17.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.780+0000 7fca41ffb640 1 -- 172.21.15.79:0/1167371856 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fca30004f60 con 0x7fca44101d80 2024-06-01T00:56:17.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.780+0000 7fca41ffb640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fca3406aef0 0x7fca3406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:17.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.780+0000 7fca41ffb640 1 -- 172.21.15.79:0/1167371856 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fca30089a60 con 0x7fca44101d80 2024-06-01T00:56:17.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.780+0000 7fca489c5640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fca3406aef0 0x7fca3406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:17.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.781+0000 7fca489c5640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fca3406aef0 0x7fca3406d3b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca38009230 tx=0x7fca38005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:17.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.784+0000 7fca41ffb640 1 -- 172.21.15.79:0/1167371856 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fca30055040 con 0x7fca44101d80 2024-06-01T00:56:17.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.996+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fca4404e210 con 0x7fca44101d80 2024-06-01T00:56:17.997 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.996+0000 7fca41ffb640 1 -- 172.21.15.79:0/1167371856 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fca3001d0d0 con 0x7fca44101d80 2024-06-01T00:56:17.997 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:17.997 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fca3406aef0 msgr2=0x7fca3406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fca3406aef0 0x7fca3406d3b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca38009230 tx=0x7fca38005a20 comp rx=0 tx=0).stop 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101d80 msgr2=0x7fca440ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101d80 0x7fca440ff600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fca3000c8f0 tx=0x7fca3000cdb0 comp rx=0 tx=0).stop 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 shutdown_connections 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fca3406aef0 0x7fca3406d3b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fca44101d80 0x7fca440ff600 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 --2- 172.21.15.79:0/1167371856 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fca44101460 0x7fca440ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 >> 172.21.15.79:0/1167371856 conn(0x7fca44075e10 msgr2=0x7fca441026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 shutdown_connections 2024-06-01T00:56:17.999 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:17.998+0000 7fca4ac29640 1 -- 172.21.15.79:0/1167371856 wait complete. 2024-06-01T00:56:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:18 smithi160 ceph-mon[36639]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:18.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:18 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1167371856' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:18.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:18 smithi079 ceph-mon[29356]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:18.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:18 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1167371856' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:20.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:20 smithi079 ceph-mon[29356]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:20.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:20 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:20 smithi160 ceph-mon[36639]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:20.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:20 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:21.676 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:21.892 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:22.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:22 smithi079 ceph-mon[29356]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:22.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:22 smithi160 ceph-mon[36639]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:23.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2383191151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 msgr2=0x7fb8a8104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:23.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2383191151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8104eb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009c20 tx=0x7fb89c02a970 comp rx=0 tx=0).stop 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2383191151 shutdown_connections 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2383191151 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8104eb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2383191151 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a81004b0 0x7fb8a8100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2383191151 >> 172.21.15.79:0/2383191151 conn(0x7fb8a80fbda0 msgr2=0x7fb8a80fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2383191151 shutdown_connections 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2383191151 wait complete. 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.600+0000 7fb8ad7ec640 1 Processor -- start 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8ad7ec640 1 -- start start 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8ad7ec640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a81004b0 0x7fb8a80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8ad7ec640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8ad7ec640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8073230 con 0x7fb8a8100e60 2024-06-01T00:56:23.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8ad7ec640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a80733a0 con 0x7fb8a81004b0 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a67fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a67fc640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8072cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33662/0 (socket says 172.21.15.79:33662) 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a67fc640 1 -- 172.21.15.79:0/2559904751 learned_addr learned my addr 172.21.15.79:0/2559904751 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a6ffd640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a81004b0 0x7fb8a80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a67fc640 1 -- 172.21.15.79:0/2559904751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a81004b0 msgr2=0x7fb8a80727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a67fc640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a81004b0 0x7fb8a80727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a67fc640 1 -- 172.21.15.79:0/2559904751 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c009900 con 0x7fb8a8100e60 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.601+0000 7fb8a67fc640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8072cf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb89c02ae90 tx=0x7fb89c030880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.602+0000 7fb88ffff640 1 -- 172.21.15.79:0/2559904751 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89c02f3c0 con 0x7fb8a8100e60 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.602+0000 7fb88ffff640 1 -- 172.21.15.79:0/2559904751 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb89c02f9e0 con 0x7fb8a8100e60 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.602+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a8073620 con 0x7fb8a8100e60 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.602+0000 7fb88ffff640 1 -- 172.21.15.79:0/2559904751 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89c039450 con 0x7fb8a8100e60 2024-06-01T00:56:23.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.602+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a81a2670 con 0x7fb8a8100e60 2024-06-01T00:56:23.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.603+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a8101990 con 0x7fb8a8100e60 2024-06-01T00:56:23.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.603+0000 7fb88ffff640 1 -- 172.21.15.79:0/2559904751 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb89c00a040 con 0x7fb8a8100e60 2024-06-01T00:56:23.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.604+0000 7fb88ffff640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb89006aef0 0x7fb89006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:23.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.604+0000 7fb88ffff640 1 -- 172.21.15.79:0/2559904751 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb89c02c040 con 0x7fb8a8100e60 2024-06-01T00:56:23.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.604+0000 7fb8a6ffd640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb89006aef0 0x7fb89006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:23.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.604+0000 7fb8a6ffd640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb89006aef0 0x7fb89006d3b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0005fd0 tx=0x7fb8a000b390 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:23.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.608+0000 7fb88ffff640 1 -- 172.21.15.79:0/2559904751 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb89c076450 con 0x7fb8a8100e60 2024-06-01T00:56:23.817 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:23.818 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:23.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.816+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8a8073620 con 0x7fb8a8100e60 2024-06-01T00:56:23.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.816+0000 7fb88ffff640 1 -- 172.21.15.79:0/2559904751 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb8a8073620 con 0x7fb8a8100e60 2024-06-01T00:56:23.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb89006aef0 msgr2=0x7fb89006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:23.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb89006aef0 0x7fb89006d3b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0005fd0 tx=0x7fb8a000b390 comp rx=0 tx=0).stop 2024-06-01T00:56:23.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 msgr2=0x7fb8a8072cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:23.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8072cf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb89c02ae90 tx=0x7fb89c030880 comp rx=0 tx=0).stop 2024-06-01T00:56:23.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 shutdown_connections 2024-06-01T00:56:23.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb89006aef0 0x7fb89006d3b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:23.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8a8100e60 0x7fb8a8072cf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:23.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 --2- 172.21.15.79:0/2559904751 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb8a81004b0 0x7fb8a80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:23.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.818+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 >> 172.21.15.79:0/2559904751 conn(0x7fb8a80fbda0 msgr2=0x7fb8a8103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:23.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.819+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 shutdown_connections 2024-06-01T00:56:23.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:23.819+0000 7fb8ad7ec640 1 -- 172.21.15.79:0/2559904751 wait complete. 2024-06-01T00:56:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:24 smithi160 ceph-mon[36639]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:24.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:24 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2559904751' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:24.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:24 smithi079 ceph-mon[29356]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:24.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:24 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2559904751' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:26.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:26 smithi160 ceph-mon[36639]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:26.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:26 smithi079 ceph-mon[29356]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:27.515 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:27.731 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:28.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:28 smithi160 ceph-mon[36639]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:28.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:28 smithi079 ceph-mon[29356]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 -- 172.21.15.79:0/821445068 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101d20 msgr2=0x7f309c10a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 --2- 172.21.15.79:0/821445068 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101d20 0x7f309c10a210 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3084009880 tx=0x7f308402a530 comp rx=0 tx=0).stop 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 -- 172.21.15.79:0/821445068 shutdown_connections 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 --2- 172.21.15.79:0/821445068 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101d20 0x7f309c10a210 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 --2- 172.21.15.79:0/821445068 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f309c101400 0x7f309c1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 -- 172.21.15.79:0/821445068 >> 172.21.15.79:0/821445068 conn(0x7f309c0fac50 msgr2=0x7f309c0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 -- 172.21.15.79:0/821445068 shutdown_connections 2024-06-01T00:56:29.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.347+0000 7f30a2591640 1 -- 172.21.15.79:0/821445068 wait complete. 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f30a2591640 1 Processor -- start 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f30a2591640 1 -- start start 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f30a2591640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101400 0x7f309c196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f30a2591640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f309c101d20 0x7f309c1967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f30a2591640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c196e30 con 0x7f309c101400 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f30a2591640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c19ab50 con 0x7f309c101d20 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f309bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101400 0x7f309c196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f309bfff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101400 0x7f309c196260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:33668/0 (socket says 172.21.15.79:33668) 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f309bfff640 1 -- 172.21.15.79:0/2936915957 learned_addr learned my addr 172.21.15.79:0/2936915957 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f309bfff640 1 -- 172.21.15.79:0/2936915957 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f309c101d20 msgr2=0x7f309c1967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f309b7fe640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f309c101d20 0x7f309c1967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f309bfff640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f309c101d20 0x7f309c1967a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:29.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.348+0000 7f309bfff640 1 -- 172.21.15.79:0/2936915957 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084009560 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.349+0000 7f309bfff640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101400 0x7f309c196260 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c810 tx=0x7f308c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.349+0000 7f30997fa640 1 -- 172.21.15.79:0/2936915957 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308c0041f0 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.349+0000 7f30997fa640 1 -- 172.21.15.79:0/2936915957 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f308c007e70 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.349+0000 7f30997fa640 1 -- 172.21.15.79:0/2936915957 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308c0049d0 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.349+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c19ae30 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.349+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c19b330 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.350+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c069080 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.350+0000 7f30997fa640 1 -- 172.21.15.79:0/2936915957 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f308c0079b0 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.351+0000 7f30997fa640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f308806aef0 0x7f308806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.351+0000 7f30997fa640 1 -- 172.21.15.79:0/2936915957 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f308c089a50 con 0x7f309c101400 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.351+0000 7f309b7fe640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f308806aef0 0x7f308806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.351+0000 7f309b7fe640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f308806aef0 0x7f308806d3b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3084000c00 tx=0x7f3084004120 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:29.365 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.355+0000 7f30997fa640 1 -- 172.21.15.79:0/2936915957 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f308c055030 con 0x7f309c101400 2024-06-01T00:56:29.567 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.567+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f309c04e210 con 0x7f309c101400 2024-06-01T00:56:29.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.567+0000 7f30997fa640 1 -- 172.21.15.79:0/2936915957 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f308c01d0d0 con 0x7f309c101400 2024-06-01T00:56:29.568 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:29.569 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:29.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f308806aef0 msgr2=0x7f308806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:29.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f308806aef0 0x7f308806d3b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3084000c00 tx=0x7f3084004120 comp rx=0 tx=0).stop 2024-06-01T00:56:29.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101400 msgr2=0x7f309c196260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:29.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101400 0x7f309c196260 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f308c00c810 tx=0x7f308c00ccd0 comp rx=0 tx=0).stop 2024-06-01T00:56:29.570 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 shutdown_connections 2024-06-01T00:56:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f308806aef0 0x7f308806d3b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f309c101d20 0x7f309c1967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.569+0000 7f30a2591640 1 --2- 172.21.15.79:0/2936915957 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f309c101400 0x7f309c196260 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.570+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 >> 172.21.15.79:0/2936915957 conn(0x7f309c0fac50 msgr2=0x7f309c1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.570+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 shutdown_connections 2024-06-01T00:56:29.571 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:29.570+0000 7f30a2591640 1 -- 172.21.15.79:0/2936915957 wait complete. 2024-06-01T00:56:30.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:30 smithi079 ceph-mon[29356]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:30.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:30 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/2936915957' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:30.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:30 smithi160 ceph-mon[36639]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:30.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:30 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/2936915957' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:32.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:32 smithi160 ceph-mon[36639]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:32.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:32 smithi079 ceph-mon[29356]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:33.340 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:33.559 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:34.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:34 smithi160 ceph-mon[36639]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:34.781 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:34 smithi079 ceph-mon[29356]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:35.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 -- 172.21.15.79:0/407936157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100460 msgr2=0x7f7224100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:35.268 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 --2- 172.21.15.79:0/407936157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100460 0x7f7224100840 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7220009880 tx=0x7f722002a4a0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 -- 172.21.15.79:0/407936157 shutdown_connections 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 --2- 172.21.15.79:0/407936157 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7224100e10 0x7f7224104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 --2- 172.21.15.79:0/407936157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100460 0x7f7224100840 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 -- 172.21.15.79:0/407936157 >> 172.21.15.79:0/407936157 conn(0x7f72240fbd10 msgr2=0x7f72240fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 -- 172.21.15.79:0/407936157 shutdown_connections 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.263+0000 7f722c52d640 1 -- 172.21.15.79:0/407936157 wait complete. 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f722c52d640 1 Processor -- start 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f722c52d640 1 -- start start 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f722c52d640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7224100460 0x7f7224196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f722c52d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100e10 0x7f7224196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:35.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f722c52d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7224196ed0 con 0x7f7224100e10 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f722c52d640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722419abf0 con 0x7f7224100460 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f7229ac8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100e10 0x7f7224196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f7229ac8640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100e10 0x7f7224196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49210/0 (socket says 172.21.15.79:49210) 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f7229ac8640 1 -- 172.21.15.79:0/1742375303 learned_addr learned my addr 172.21.15.79:0/1742375303 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f7229ac8640 1 -- 172.21.15.79:0/1742375303 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7224100460 msgr2=0x7f7224196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f7229ac8640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7224100460 0x7f7224196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f7229ac8640 1 -- 172.21.15.79:0/1742375303 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7220009560 con 0x7f7224100e10 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f7229ac8640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100e10 0x7f7224196840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f721400b3d0 tx=0x7f721400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.264+0000 7f721b7fe640 1 -- 172.21.15.79:0/1742375303 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7214011800 con 0x7f7224100e10 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.265+0000 7f721b7fe640 1 -- 172.21.15.79:0/1742375303 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7214011e20 con 0x7f7224100e10 2024-06-01T00:56:35.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.265+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722419aed0 con 0x7f7224100e10 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.265+0000 7f721b7fe640 1 -- 172.21.15.79:0/1742375303 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f721400f2e0 con 0x7f7224100e10 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.265+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722419b420 con 0x7f7224100e10 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.266+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224101940 con 0x7f7224100e10 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.266+0000 7f721b7fe640 1 -- 172.21.15.79:0/1742375303 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f7214010380 con 0x7f7224100e10 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.267+0000 7f721b7fe640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f721006aef0 0x7f721006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.267+0000 7f721b7fe640 1 -- 172.21.15.79:0/1742375303 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f72140897e0 con 0x7f7224100e10 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.267+0000 7f722a2c9640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f721006aef0 0x7f721006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:35.271 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.267+0000 7f722a2c9640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f721006aef0 0x7f721006d3b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7220009880 tx=0x7f7220000bc0 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:35.272 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.272+0000 7f721b7fe640 1 -- 172.21.15.79:0/1742375303 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f7214054e40 con 0x7f7224100e10 2024-06-01T00:56:35.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.486+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f722404e210 con 0x7f7224100e10 2024-06-01T00:56:35.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.486+0000 7f721b7fe640 1 -- 172.21.15.79:0/1742375303 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7214058460 con 0x7f7224100e10 2024-06-01T00:56:35.487 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:35.487 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:35.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f721006aef0 msgr2=0x7f721006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:35.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f721006aef0 0x7f721006d3b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7220009880 tx=0x7f7220000bc0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100e10 msgr2=0x7f7224196840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:35.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100e10 0x7f7224196840 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f721400b3d0 tx=0x7f721400b890 comp rx=0 tx=0).stop 2024-06-01T00:56:35.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 shutdown_connections 2024-06-01T00:56:35.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f721006aef0 0x7f721006d3b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7224100e10 0x7f7224196840 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 --2- 172.21.15.79:0/1742375303 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f7224100460 0x7f7224196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:35.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.488+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 >> 172.21.15.79:0/1742375303 conn(0x7f72240fbd10 msgr2=0x7f72240fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:35.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.489+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 shutdown_connections 2024-06-01T00:56:35.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:35.489+0000 7f722c52d640 1 -- 172.21.15.79:0/1742375303 wait complete. 2024-06-01T00:56:35.789 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:35 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:35.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:35 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:36 smithi160 ceph-mon[36639]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:36.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:36 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1742375303' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:36.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:36 smithi079 ceph-mon[29356]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:36.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:36 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1742375303' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:38.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:38 smithi160 ceph-mon[36639]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:38.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:38 smithi079 ceph-mon[29356]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:39.070 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:39.287 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:40.764 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:40 smithi079 ceph-mon[29356]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:40.807 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:40 smithi160 ceph-mon[36639]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:41.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1156624663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e40fe9f0 msgr2=0x7fb1e40fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:41.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1156624663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e40fe9f0 0x7fb1e40fee30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400a990 tx=0x7fb1d402b7d0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1156624663 shutdown_connections 2024-06-01T00:56:41.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1156624663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e40fe9f0 0x7fb1e40fee30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1156624663 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1e41048e0 0x7fb1e4104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.011 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1156624663 >> 172.21.15.79:0/1156624663 conn(0x7fb1e4074430 msgr2=0x7fb1e4074840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1156624663 shutdown_connections 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.994+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1156624663 wait complete. 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1eafa6640 1 Processor -- start 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1eafa6640 1 -- start start 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1eafa6640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1e40fe9f0 0x7fb1e41962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1eafa6640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e41048e0 0x7fb1e41967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1eafa6640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e4196e80 con 0x7fb1e41048e0 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1eafa6640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e419aba0 con 0x7fb1e40fe9f0 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1e3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e41048e0 0x7fb1e41967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1e3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e41048e0 0x7fb1e41967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49230/0 (socket says 172.21.15.79:49230) 2024-06-01T00:56:41.012 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.995+0000 7fb1e3fff640 1 -- 172.21.15.79:0/1312552301 learned_addr learned my addr 172.21.15.79:0/1312552301 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1e3fff640 1 -- 172.21.15.79:0/1312552301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1e40fe9f0 msgr2=0x7fb1e41962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1e3fff640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1e40fe9f0 0x7fb1e41962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1e3fff640 1 -- 172.21.15.79:0/1312552301 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d400a670 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1e3fff640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e41048e0 0x7fb1e41967f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb1d402bcf0 tx=0x7fb1d40048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1e1ffb640 1 -- 172.21.15.79:0/1312552301 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d4004420 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1e1ffb640 1 -- 172.21.15.79:0/1312552301 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1d4004ae0 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e419ae20 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1e1ffb640 1 -- 172.21.15.79:0/1312552301 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1d40053b0 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.996+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e419b370 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.997+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e4100020 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.997+0000 7fb1e1ffb640 1 -- 172.21.15.79:0/1312552301 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fb1d40035f0 con 0x7fb1e41048e0 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.998+0000 7fb1e1ffb640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb1d006aef0 0x7fb1d006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:41.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.998+0000 7fb1e1ffb640 1 -- 172.21.15.79:0/1312552301 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb1d40ab2f0 con 0x7fb1e41048e0 2024-06-01T00:56:41.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.998+0000 7fb1e8d42640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb1d006aef0 0x7fb1d006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:41.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:40.998+0000 7fb1e8d42640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb1d006aef0 0x7fb1d006d3b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009710 tx=0x7fb1cc006c90 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:41.014 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.002+0000 7fb1e1ffb640 1 -- 172.21.15.79:0/1312552301 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fb1d4076a70 con 0x7fb1e41048e0 2024-06-01T00:56:41.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.214+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb1e4108e10 con 0x7fb1e41048e0 2024-06-01T00:56:41.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.215+0000 7fb1e1ffb640 1 -- 172.21.15.79:0/1312552301 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb1d407a090 con 0x7fb1e41048e0 2024-06-01T00:56:41.217 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:41.217 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:41.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb1d006aef0 msgr2=0x7fb1d006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:41.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb1d006aef0 0x7fb1d006d3b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009710 tx=0x7fb1cc006c90 comp rx=0 tx=0).stop 2024-06-01T00:56:41.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e41048e0 msgr2=0x7fb1e41967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:41.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e41048e0 0x7fb1e41967f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb1d402bcf0 tx=0x7fb1d40048f0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 shutdown_connections 2024-06-01T00:56:41.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fb1d006aef0 0x7fb1d006d3b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1e41048e0 0x7fb1e41967f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 --2- 172.21.15.79:0/1312552301 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fb1e40fe9f0 0x7fb1e41962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:41.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 >> 172.21.15.79:0/1312552301 conn(0x7fb1e4074430 msgr2=0x7fb1e40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:41.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 shutdown_connections 2024-06-01T00:56:41.220 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:41.217+0000 7fb1eafa6640 1 -- 172.21.15.79:0/1312552301 wait complete. 2024-06-01T00:56:41.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:41 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/1312552301' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:41.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:41 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/1312552301' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:42.605 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:42 smithi079 ceph-mon[29356]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:42.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:42 smithi160 ceph-mon[36639]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:44.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:44 smithi160 ceph-mon[36639]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:44.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:44 smithi079 ceph-mon[29356]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:44.918 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:45.139 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:46.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:46 smithi079 ceph-mon[29356]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:46.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 -- 172.21.15.79:0/3117449090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 msgr2=0x7f221c0fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:46.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 --2- 172.21.15.79:0/3117449090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c0fee30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2204009c20 tx=0x7f220402a970 comp rx=0 tx=0).stop 2024-06-01T00:56:46.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 -- 172.21.15.79:0/3117449090 shutdown_connections 2024-06-01T00:56:46.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 --2- 172.21.15.79:0/3117449090 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c0fee30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:46.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 --2- 172.21.15.79:0/3117449090 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f221c1062f0 0x7f221c1066d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:46.787 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 -- 172.21.15.79:0/3117449090 >> 172.21.15.79:0/3117449090 conn(0x7f221c074430 msgr2=0x7f221c074840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:46.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 -- 172.21.15.79:0/3117449090 shutdown_connections 2024-06-01T00:56:46.788 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.786+0000 7f2221043640 1 -- 172.21.15.79:0/3117449090 wait complete. 2024-06-01T00:56:46.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.787+0000 7f2221043640 1 Processor -- start 2024-06-01T00:56:46.789 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.787+0000 7f2221043640 1 -- start start 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.787+0000 7f2221043640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.787+0000 7f2221043640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f221c1062f0 0x7f221c196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.787+0000 7f2221043640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c196ef0 con 0x7f221c0fe9f0 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.787+0000 7f2221043640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c19ac10 con 0x7f221c1062f0 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.787+0000 7f221ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f221ad9d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48776/0 (socket says 172.21.15.79:48776) 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f221ad9d640 1 -- 172.21.15.79:0/3312345789 learned_addr learned my addr 172.21.15.79:0/3312345789 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f221a59c640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f221c1062f0 0x7f221c196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f221ad9d640 1 -- 172.21.15.79:0/3312345789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f221c1062f0 msgr2=0x7f221c196860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f221ad9d640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f221c1062f0 0x7f221c196860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f221ad9d640 1 -- 172.21.15.79:0/3312345789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204009900 con 0x7f221c0fe9f0 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f221ad9d640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c196320 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f221400b420 tx=0x7f221400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:46.790 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f2213fff640 1 -- 172.21.15.79:0/3312345789 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2214011800 con 0x7f221c0fe9f0 2024-06-01T00:56:46.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221c19aef0 con 0x7f221c0fe9f0 2024-06-01T00:56:46.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f2213fff640 1 -- 172.21.15.79:0/3312345789 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2214011e20 con 0x7f221c0fe9f0 2024-06-01T00:56:46.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f2213fff640 1 -- 172.21.15.79:0/3312345789 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f221400f2e0 con 0x7f221c0fe9f0 2024-06-01T00:56:46.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.788+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221c19b440 con 0x7f221c0fe9f0 2024-06-01T00:56:46.791 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.789+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221c068410 con 0x7f221c0fe9f0 2024-06-01T00:56:46.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.790+0000 7f2213fff640 1 -- 172.21.15.79:0/3312345789 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f2214010380 con 0x7f221c0fe9f0 2024-06-01T00:56:46.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.790+0000 7f2213fff640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f220806aef0 0x7f220806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:46.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.790+0000 7f221a59c640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f220806aef0 0x7f220806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:46.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.790+0000 7f2213fff640 1 -- 172.21.15.79:0/3312345789 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f22140894e0 con 0x7f221c0fe9f0 2024-06-01T00:56:46.792 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.791+0000 7f221a59c640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f220806aef0 0x7f220806d3b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f221c197880 tx=0x7f220403d040 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:46.795 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:46.795+0000 7f2213fff640 1 -- 172.21.15.79:0/3312345789 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f2214054ac0 con 0x7f221c0fe9f0 2024-06-01T00:56:46.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:46 smithi160 ceph-mon[36639]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:47.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.004+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f221c04e210 con 0x7f221c0fe9f0 2024-06-01T00:56:47.005 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.004+0000 7f2213fff640 1 -- 172.21.15.79:0/3312345789 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f22140580e0 con 0x7f221c0fe9f0 2024-06-01T00:56:47.005 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:47.005 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]} 2024-06-01T00:56:47.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f220806aef0 msgr2=0x7f220806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:47.007 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f220806aef0 0x7f220806d3b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f221c197880 tx=0x7f220403d040 comp rx=0 tx=0).stop 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 msgr2=0x7f221c196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c196320 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f221400b420 tx=0x7f221400b8e0 comp rx=0 tx=0).stop 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 shutdown_connections 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f220806aef0 0x7f220806d3b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f221c1062f0 0x7f221c196860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 --2- 172.21.15.79:0/3312345789 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f221c0fe9f0 0x7f221c196320 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.006+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 >> 172.21.15.79:0/3312345789 conn(0x7f221c074430 msgr2=0x7f221c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.007+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 shutdown_connections 2024-06-01T00:56:47.008 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:47.007+0000 7f2221043640 1 -- 172.21.15.79:0/3312345789 wait complete. 2024-06-01T00:56:47.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:47 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/3312345789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:47.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:47 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/3312345789' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:48.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:48 smithi079 ceph-mon[29356]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:48.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:48 smithi160 ceph-mon[36639]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:49.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:49 smithi160 ceph-mon[36639]: from='client.14636 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:56:49.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:49 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:56:49.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:49 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:56:49.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:49 smithi079 ceph-mon[29356]: from='client.14636 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:56:49.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:49 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:56:49.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:49 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' 2024-06-01T00:56:50.717 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:50 smithi160 ceph-mon[36639]: from='client.14640 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:56:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:50 smithi160 ceph-mon[36639]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:50 smithi160 ceph-mon[36639]: from='client.14644 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:56:50.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:50 smithi160 ceph-mon[36639]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:50.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:50 smithi160 ceph-mon[36639]: Health check update: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-01T00:56:50.805 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:50 smithi160 ceph-mon[36639]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:50.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:50 smithi079 ceph-mon[29356]: from='client.14640 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:56:50.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:50 smithi079 ceph-mon[29356]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:50.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:50 smithi079 ceph-mon[29356]: from='client.14644 -' entity='client.crash.smithi079' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T00:56:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:50 smithi079 ceph-mon[29356]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:50 smithi079 ceph-mon[29356]: Health check update: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-01T00:56:50.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:50 smithi079 ceph-mon[29356]: from='mgr.14215 172.21.15.79:0/1253636021' entity='mgr.smithi079.emylav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T00:56:50.926 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:52.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.600+0000 7f1c62aae640 1 -- 172.21.15.79:0/1480670720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100e20 msgr2=0x7f1c5c104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:52.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.600+0000 7f1c62aae640 1 --2- 172.21.15.79:0/1480670720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100e20 0x7f1c5c104e70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009880 tx=0x7f1c4c02a4a0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.600+0000 7f1c62aae640 1 -- 172.21.15.79:0/1480670720 shutdown_connections 2024-06-01T00:56:52.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.600+0000 7f1c62aae640 1 --2- 172.21.15.79:0/1480670720 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100e20 0x7f1c5c104e70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.600+0000 7f1c62aae640 1 --2- 172.21.15.79:0/1480670720 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c5c100470 0x7f1c5c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.600+0000 7f1c62aae640 1 -- 172.21.15.79:0/1480670720 >> 172.21.15.79:0/1480670720 conn(0x7f1c5c0fbd20 msgr2=0x7f1c5c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 -- 172.21.15.79:0/1480670720 shutdown_connections 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 -- 172.21.15.79:0/1480670720 wait complete. 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 Processor -- start 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 -- start start 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100470 0x7f1c5c196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c5c100e20 0x7f1c5c196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c196e20 con 0x7f1c5c100470 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c62aae640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c19ab40 con 0x7f1c5c100e20 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c6084a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100470 0x7f1c5c196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c6084a640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100470 0x7f1c5c196250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47616/0 (socket says 172.21.15.79:47616) 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.601+0000 7f1c6084a640 1 -- 172.21.15.79:0/4067975289 learned_addr learned my addr 172.21.15.79:0/4067975289 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c53fff640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c5c100e20 0x7f1c5c196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:52.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c6084a640 1 -- 172.21.15.79:0/4067975289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c5c100e20 msgr2=0x7f1c5c196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c6084a640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c5c100e20 0x7f1c5c196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c6084a640 1 -- 172.21.15.79:0/4067975289 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c009560 con 0x7f1c5c100470 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c6084a640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100470 0x7f1c5c196250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800c8f0 tx=0x7f1c5800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c51ffb640 1 -- 172.21.15.79:0/4067975289 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c58007b80 con 0x7f1c5c100470 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c51ffb640 1 -- 172.21.15.79:0/4067975289 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c580054e0 con 0x7f1c5c100470 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5c19ae20 con 0x7f1c5c100470 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c51ffb640 1 -- 172.21.15.79:0/4067975289 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c5800f450 con 0x7f1c5c100470 2024-06-01T00:56:52.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.602+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5c19b370 con 0x7f1c5c100470 2024-06-01T00:56:52.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.603+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5c101950 con 0x7f1c5c100470 2024-06-01T00:56:52.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.603+0000 7f1c51ffb640 1 -- 172.21.15.79:0/4067975289 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7f1c58004f60 con 0x7f1c5c100470 2024-06-01T00:56:52.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.604+0000 7f1c51ffb640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1c4406aef0 0x7f1c4406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:52.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.604+0000 7f1c51ffb640 1 -- 172.21.15.79:0/4067975289 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1c58089a60 con 0x7f1c5c100470 2024-06-01T00:56:52.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.604+0000 7f1c53fff640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1c4406aef0 0x7f1c4406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:52.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.604+0000 7f1c53fff640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1c4406aef0 0x7f1c4406d3b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c000c00 tx=0x7f1c4c004120 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:52.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.608+0000 7f1c51ffb640 1 -- 172.21.15.79:0/4067975289 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7f1c58055040 con 0x7f1c5c100470 2024-06-01T00:56:52.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:52 smithi160 ceph-mon[36639]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:52.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.815+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1c5c04e210 con 0x7f1c5c100470 2024-06-01T00:56:52.816 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:52 smithi079 ceph-mon[29356]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:52.816 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.815+0000 7f1c51ffb640 1 -- 172.21.15.79:0/4067975289 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1c5801d0d0 con 0x7f1c5c100470 2024-06-01T00:56:52.816 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:52.816 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1c4406aef0 msgr2=0x7f1c4406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1c4406aef0 0x7f1c4406d3b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c000c00 tx=0x7f1c4c004120 comp rx=0 tx=0).stop 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100470 msgr2=0x7f1c5c196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100470 0x7f1c5c196250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800c8f0 tx=0x7f1c5800cdb0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 shutdown_connections 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7f1c4406aef0 0x7f1c4406d3b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7f1c5c100e20 0x7f1c5c196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 --2- 172.21.15.79:0/4067975289 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1c5c100470 0x7f1c5c196250 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 >> 172.21.15.79:0/4067975289 conn(0x7f1c5c0fbd20 msgr2=0x7f1c5c103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:52.818 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 shutdown_connections 2024-06-01T00:56:52.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:52.817+0000 7f1c62aae640 1 -- 172.21.15.79:0/4067975289 wait complete. 2024-06-01T00:56:53.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:53 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4067975289' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:53.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:53 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4067975289' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:54.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:54 smithi160 ceph-mon[36639]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:54.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:54 smithi079 ceph-mon[29356]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:56.532 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T00:56:56.748 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/mon.smithi079/config 2024-06-01T00:56:56.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:56 smithi160 ceph-mon[36639]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:56.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:56 smithi079 ceph-mon[29356]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:58.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- 172.21.15.79:0/4150867449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101d60 msgr2=0x7fd87c10a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:58.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 --2- 172.21.15.79:0/4150867449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101d60 0x7fd87c10a250 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009880 tx=0x7fd86c02a530 comp rx=0 tx=0).stop 2024-06-01T00:56:58.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- 172.21.15.79:0/4150867449 shutdown_connections 2024-06-01T00:56:58.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 --2- 172.21.15.79:0/4150867449 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101d60 0x7fd87c10a250 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:58.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 --2- 172.21.15.79:0/4150867449 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd87c101440 0x7fd87c101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:58.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- 172.21.15.79:0/4150867449 >> 172.21.15.79:0/4150867449 conn(0x7fd87c075e10 msgr2=0x7fd87c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- 172.21.15.79:0/4150867449 shutdown_connections 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- 172.21.15.79:0/4150867449 wait complete. 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 Processor -- start 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- start start 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101440 0x7fd87c196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 --2- >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd87c101d60 0x7fd87c196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c196dd0 con 0x7fd87c101440 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.457+0000 7fd884321640 1 -- --> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c19aaf0 con 0x7fd87c101d60 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8820bd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101440 0x7fd87c196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8820bd640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101440 0x7fd87c196200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:47626/0 (socket says 172.21.15.79:47626) 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8820bd640 1 -- 172.21.15.79:0/4131124361 learned_addr learned my addr 172.21.15.79:0/4131124361 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-06-01T00:56:58.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8818bc640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd87c101d60 0x7fd87c196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8820bd640 1 -- 172.21.15.79:0/4131124361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd87c101d60 msgr2=0x7fd87c196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8820bd640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd87c101d60 0x7fd87c196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8820bd640 1 -- 172.21.15.79:0/4131124361 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c009560 con 0x7fd87c101440 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8820bd640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101440 0x7fd87c196200 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd87400cc60 tx=0x7fd87400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8737fe640 1 -- 172.21.15.79:0/4131124361 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd874004b20 con 0x7fd87c101440 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c19add0 con 0x7fd87c101440 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8737fe640 1 -- 172.21.15.79:0/4131124361 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd874004c80 con 0x7fd87c101440 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd8737fe640 1 -- 172.21.15.79:0/4131124361 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd87400f640 con 0x7fd87c101440 2024-06-01T00:56:58.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.458+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c19b2f0 con 0x7fd87c101440 2024-06-01T00:56:58.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.459+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c068490 con 0x7fd87c101440 2024-06-01T00:56:58.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.460+0000 7fd8737fe640 1 -- 172.21.15.79:0/4131124361 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 89470+0+0 (secure 0 0 0) 0x7fd8740055e0 con 0x7fd87c101440 2024-06-01T00:56:58.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.460+0000 7fd8737fe640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd86806aef0 0x7fd86806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-01T00:56:58.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.460+0000 7fd8737fe640 1 -- 172.21.15.79:0/4131124361 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd874089db0 con 0x7fd87c101440 2024-06-01T00:56:58.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.460+0000 7fd8818bc640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd86806aef0 0x7fd86806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-01T00:56:58.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.461+0000 7fd8818bc640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd86806aef0 0x7fd86806d3b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd86c000c00 tx=0x7fd86c005a20 comp rx=0 tx=0).ready entity=mgr.14215 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-01T00:56:58.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.464+0000 7fd8737fe640 1 -- 172.21.15.79:0/4131124361 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (secure 0 0 0) 0x7fd874055390 con 0x7fd87c101440 2024-06-01T00:56:58.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.672+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd87c04e210 con 0x7fd87c101440 2024-06-01T00:56:58.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.673+0000 7fd8737fe640 1 -- 172.21.15.79:0/4131124361 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd8740589b0 con 0x7fd87c101440 2024-06-01T00:56:58.674 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-06-01T00:56:58.674 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-01T00:56:58.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.674+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd86806aef0 msgr2=0x7fd86806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:58.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.674+0000 7fd884321640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd86806aef0 0x7fd86806d3b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd86c000c00 tx=0x7fd86c005a20 comp rx=0 tx=0).stop 2024-06-01T00:56:58.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101440 msgr2=0x7fd87c196200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-01T00:56:58.675 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101440 0x7fd87c196200 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd87400cc60 tx=0x7fd87400a620 comp rx=0 tx=0).stop 2024-06-01T00:56:58.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 shutdown_connections 2024-06-01T00:56:58.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:6800/187237089,v1:172.21.15.79:6801/187237089] conn(0x7fd86806aef0 0x7fd86806d3b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:58.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] conn(0x7fd87c101d60 0x7fd87c196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:58.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 --2- 172.21.15.79:0/4131124361 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd87c101440 0x7fd87c196200 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-01T00:56:58.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 >> 172.21.15.79:0/4131124361 conn(0x7fd87c075e10 msgr2=0x7fd87c1026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-01T00:56:58.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 shutdown_connections 2024-06-01T00:56:58.676 INFO:teuthology.orchestra.run.smithi079.stderr:2024-06-01T00:56:58.675+0000 7fd884321640 1 -- 172.21.15.79:0/4131124361 wait complete. 2024-06-01T00:56:58.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:58 smithi160 ceph-mon[36639]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:58.855 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:58 smithi079 ceph-mon[29356]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-06-01T00:56:59.355 INFO:tasks.cephadm:Teardown begin 2024-06-01T00:56:59.355 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_036ad575e36eb2f5d36bafd4f61b4922eadba901/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-01T00:56:59.358 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T00:56:59.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T00:56:59.422 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-01T00:56:59.422 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-01T00:56:59.442 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-01T00:56:59.480 INFO:tasks.cephadm:Stopping all daemons... 2024-06-01T00:56:59.481 INFO:tasks.cephadm.mon.smithi079:Stopping mon.smithi079... 2024-06-01T00:56:59.481 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079 2024-06-01T00:56:59.787 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:59 smithi079 systemd[1]: Stopping Ceph mon.smithi079 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:56:59.787 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:59 smithi079 ceph-mon[29356]: from='client.? 172.21.15.79:0/4131124361' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:56:59.804 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:56:59 smithi160 ceph-mon[36639]: from='client.? 172.21.15.79:0/4131124361' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T00:57:00.105 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:59 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079[29352]: 2024-06-01T00:56:59.786+0000 7fef2241e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi079 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:00.105 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:59 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079[29352]: 2024-06-01T00:56:59.786+0000 7fef2241e640 -1 mon.smithi079@0(leader) e2 *** Got Signal Terminated *** 2024-06-01T00:57:00.105 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:56:59 smithi079 podman[81551]: 2024-06-01 00:56:59.908962689 +0000 UTC m=+0.230155288 container died 9cf2014362517e1e6223e305a09209c1cd79feaa8b32c8f5b5bb0f42500b2f5e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, RELEASE=reef-71c5cfc, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-01T00:57:00.764 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:57:00 smithi079 podman[81551]: 2024-06-01 00:57:00.453768829 +0000 UTC m=+0.774961419 container cleanup 9cf2014362517e1e6223e305a09209c1cd79feaa8b32c8f5b5bb0f42500b2f5e (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-71c5cfc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b) 2024-06-01T00:57:00.764 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jun 01 00:57:00 smithi079 bash[81551]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi079 2024-06-01T00:57:00.914 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi079.service' 2024-06-01T00:57:00.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:00.963 INFO:tasks.cephadm.mon.smithi079:Stopped mon.smithi079 2024-06-01T00:57:00.963 INFO:tasks.cephadm.mon.smithi160:Stopping mon.smithi160... 2024-06-01T00:57:00.964 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi160 2024-06-01T00:57:01.297 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:57:01 smithi160 systemd[1]: Stopping Ceph mon.smithi160 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:57:01.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:57:01 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160[36614]: 2024-06-01T00:57:01.296+0000 7f4d05b91640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi160 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:01.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:57:01 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160[36614]: 2024-06-01T00:57:01.296+0000 7f4d05b91640 -1 mon.smithi160@1(peon) e2 *** Got Signal Terminated *** 2024-06-01T00:57:01.554 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:57:01 smithi160 podman[60737]: 2024-06-01 00:57:01.307526354 +0000 UTC m=+0.140335511 container died 52e1af6da32d65a59cb9693cb668b833294d0eb681e880c85210aa802316192d (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-71c5cfc, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T00:57:02.022 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:57:01 smithi160 podman[60737]: 2024-06-01 00:57:01.771900503 +0000 UTC m=+0.604709705 container cleanup 52e1af6da32d65a59cb9693cb668b833294d0eb681e880c85210aa802316192d (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-01T00:57:02.023 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:Jun 01 00:57:01 smithi160 bash[60737]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-mon-smithi160 2024-06-01T00:57:02.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@mon.smithi160.service' 2024-06-01T00:57:02.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:02.211 INFO:tasks.cephadm.mon.smithi160:Stopped mon.smithi160 2024-06-01T00:57:02.211 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-01T00:57:02.211 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.0 2024-06-01T00:57:02.525 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:02 smithi079 systemd[1]: Stopping Ceph osd.0 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:57:02.855 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:02 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0[47958]: 2024-06-01T00:57:02.524+0000 7f409bcb7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:02.855 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:02 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0[47958]: 2024-06-01T00:57:02.524+0000 7f409bcb7640 -1 osd.0 41 *** Got signal Terminated *** 2024-06-01T00:57:02.856 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:02 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0[47958]: 2024-06-01T00:57:02.524+0000 7f409bcb7640 -1 osd.0 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T00:57:08.151 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:07 smithi079 podman[81769]: 2024-06-01 00:57:07.863210851 +0000 UTC m=+5.448418922 container died 04edd35a28a3b37246cc680071fe76f6710ffb3fc94072766627ae59b40351d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-71c5cfc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3) 2024-06-01T00:57:08.605 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:08 smithi079 podman[81769]: 2024-06-01 00:57:08.333239582 +0000 UTC m=+5.918447694 container cleanup 04edd35a28a3b37246cc680071fe76f6710ffb3fc94072766627ae59b40351d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, ceph=True) 2024-06-01T00:57:08.605 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:08 smithi079 bash[81769]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0 2024-06-01T00:57:08.955 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:08 smithi079 podman[81785]: 2024-06-01 00:57:08.609916362 +0000 UTC m=+0.746554728 container remove 04edd35a28a3b37246cc680071fe76f6710ffb3fc94072766627ae59b40351d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0, org.label-schema.build-date=20240528, RELEASE=reef-71c5cfc, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:09.213 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:08 smithi079 podman[81873]: 2024-06-01 00:57:08.858501223 +0000 UTC m=+0.017219779 image pull 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58 2024-06-01T00:57:09.214 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:08 smithi079 podman[81873]: 2024-06-01 00:57:08.994286466 +0000 UTC m=+0.153005029 container create de7231b9d60d05b4de1e35c5988c40a316c6d7179d74ae1e0a8dcbdb0432e90c (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0-deactivate, RELEASE=reef-71c5cfc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528) 2024-06-01T00:57:09.214 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:09 smithi079 podman[81873]: 2024-06-01 00:57:09.213686519 +0000 UTC m=+0.372405074 container init de7231b9d60d05b4de1e35c5988c40a316c6d7179d74ae1e0a8dcbdb0432e90c (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-01T00:57:09.473 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:09 smithi079 podman[81873]: 2024-06-01 00:57:09.216966685 +0000 UTC m=+0.375685228 container start de7231b9d60d05b4de1e35c5988c40a316c6d7179d74ae1e0a8dcbdb0432e90c (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-71c5cfc, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-01T00:57:09.473 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:09 smithi079 podman[81873]: 2024-06-01 00:57:09.29708178 +0000 UTC m=+0.455800330 container attach de7231b9d60d05b4de1e35c5988c40a316c6d7179d74ae1e0a8dcbdb0432e90c (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=reef-71c5cfc) 2024-06-01T00:57:09.809 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:09 smithi079 podman[81873]: 2024-06-01 00:57:09.472970509 +0000 UTC m=+0.631689054 container died de7231b9d60d05b4de1e35c5988c40a316c6d7179d74ae1e0a8dcbdb0432e90c (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_CLEAN=True) 2024-06-01T00:57:10.062 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jun 01 00:57:10 smithi079 podman[81896]: 2024-06-01 00:57:10.059117165 +0000 UTC m=+0.575463950 container remove de7231b9d60d05b4de1e35c5988c40a316c6d7179d74ae1e0a8dcbdb0432e90c (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=reef-71c5cfc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=) 2024-06-01T00:57:10.086 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-01T00:57:10.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:10.127 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-01T00:57:10.127 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-01T00:57:10.127 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.1 2024-06-01T00:57:10.355 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:10 smithi079 systemd[1]: Stopping Ceph osd.1 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:57:10.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:10 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1[55785]: 2024-06-01T00:57:10.478+0000 7f7145899640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:10.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:10 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1[55785]: 2024-06-01T00:57:10.478+0000 7f7145899640 -1 osd.1 41 *** Got signal Terminated *** 2024-06-01T00:57:10.855 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:10 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1[55785]: 2024-06-01T00:57:10.478+0000 7f7145899640 -1 osd.1 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T00:57:16.195 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:15 smithi079 podman[81975]: 2024-06-01 00:57:15.918929127 +0000 UTC m=+5.557817590 container died ba8559c81d8fcabc0c67e4d88755410dde42ed3b7fc93e3a71546ad3134b530f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , RELEASE=reef-71c5cfc, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-01T00:57:16.605 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:16 smithi079 podman[81975]: 2024-06-01 00:57:16.386936184 +0000 UTC m=+6.025824637 container cleanup ba8559c81d8fcabc0c67e4d88755410dde42ed3b7fc93e3a71546ad3134b530f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.35.3) 2024-06-01T00:57:16.605 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:16 smithi079 bash[81975]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1 2024-06-01T00:57:16.987 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:16 smithi079 podman[81990]: 2024-06-01 00:57:16.67200832 +0000 UTC m=+0.747631026 container remove ba8559c81d8fcabc0c67e4d88755410dde42ed3b7fc93e3a71546ad3134b530f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1, RELEASE=reef-71c5cfc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-06-01T00:57:17.258 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:16 smithi079 podman[82078]: 2024-06-01 00:57:16.890364567 +0000 UTC m=+0.017230136 image pull 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58 2024-06-01T00:57:17.258 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:17 smithi079 podman[82078]: 2024-06-01 00:57:17.006183877 +0000 UTC m=+0.133049447 container create f6e1acd4ba64f75fce5a130dae72c7939ebeb94cd56b980b21c3132871bcce68 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1-deactivate, RELEASE=reef-71c5cfc, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:17.259 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:17 smithi079 podman[82078]: 2024-06-01 00:57:17.258750509 +0000 UTC m=+0.385616079 container init f6e1acd4ba64f75fce5a130dae72c7939ebeb94cd56b980b21c3132871bcce68 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_CLEAN=True, ceph=True, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-01T00:57:17.533 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:17 smithi079 podman[82078]: 2024-06-01 00:57:17.26237132 +0000 UTC m=+0.389236896 container start f6e1acd4ba64f75fce5a130dae72c7939ebeb94cd56b980b21c3132871bcce68 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-71c5cfc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_CLEAN=True) 2024-06-01T00:57:17.533 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:17 smithi079 podman[82078]: 2024-06-01 00:57:17.333679596 +0000 UTC m=+0.460545175 container attach f6e1acd4ba64f75fce5a130dae72c7939ebeb94cd56b980b21c3132871bcce68 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-01T00:57:17.822 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jun 01 00:57:17 smithi079 podman[82101]: 2024-06-01 00:57:17.532821588 +0000 UTC m=+0.017681161 container died f6e1acd4ba64f75fce5a130dae72c7939ebeb94cd56b980b21c3132871bcce68 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3) 2024-06-01T00:57:18.224 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-01T00:57:18.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:18.263 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-01T00:57:18.263 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-01T00:57:18.263 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.2 2024-06-01T00:57:18.498 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:18 smithi079 systemd[1]: Stopping Ceph osd.2 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:57:18.855 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:18 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2[63407]: 2024-06-01T00:57:18.624+0000 7f11bebb8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:18.855 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:18 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2[63407]: 2024-06-01T00:57:18.624+0000 7f11bebb8640 -1 osd.2 41 *** Got signal Terminated *** 2024-06-01T00:57:18.855 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:18 smithi079 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2[63407]: 2024-06-01T00:57:18.624+0000 7f11bebb8640 -1 osd.2 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T00:57:24.355 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:24 smithi079 podman[82182]: 2024-06-01 00:57:24.088572602 +0000 UTC m=+5.583522845 container died 0b1e8771e3e486e3906faa33c23e8ddd65df535dbfceb7512d72eec8d04851d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, ceph=True, RELEASE=reef-71c5cfc, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.35.3) 2024-06-01T00:57:24.826 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:24 smithi079 podman[82182]: 2024-06-01 00:57:24.55785988 +0000 UTC m=+6.052810144 container cleanup 0b1e8771e3e486e3906faa33c23e8ddd65df535dbfceb7512d72eec8d04851d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, ceph=True, GIT_CLEAN=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.license=GPLv2, RELEASE=reef-71c5cfc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:24.826 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:24 smithi079 bash[82182]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2 2024-06-01T00:57:25.105 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:24 smithi079 podman[82208]: 2024-06-01 00:57:24.82597736 +0000 UTC m=+0.730164529 container remove 0b1e8771e3e486e3906faa33c23e8ddd65df535dbfceb7512d72eec8d04851d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-01T00:57:25.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:25 smithi079 podman[82300]: 2024-06-01 00:57:25.086703122 +0000 UTC m=+0.017122577 image pull 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58 2024-06-01T00:57:25.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:25 smithi079 podman[82300]: 2024-06-01 00:57:25.201774355 +0000 UTC m=+0.132193868 container create 402a02de89575c0cfe2f15b6c5ebc43758807bccc94b412f3c7a4963f6dd2233 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b) 2024-06-01T00:57:25.446 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:25 smithi079 podman[82300]: 2024-06-01 00:57:25.446515882 +0000 UTC m=+0.376935326 container init 402a02de89575c0cfe2f15b6c5ebc43758807bccc94b412f3c7a4963f6dd2233 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=reef-71c5cfc, org.label-schema.vendor=CentOS, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-01T00:57:25.704 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:25 smithi079 podman[82300]: 2024-06-01 00:57:25.450001514 +0000 UTC m=+0.380420966 container start 402a02de89575c0cfe2f15b6c5ebc43758807bccc94b412f3c7a4963f6dd2233 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=reef-71c5cfc, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:25.704 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:25 smithi079 podman[82300]: 2024-06-01 00:57:25.546289424 +0000 UTC m=+0.476708869 container attach 402a02de89575c0cfe2f15b6c5ebc43758807bccc94b412f3c7a4963f6dd2233 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, org.label-schema.vendor=CentOS, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b) 2024-06-01T00:57:25.999 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:25 smithi079 podman[82300]: 2024-06-01 00:57:25.704947214 +0000 UTC m=+0.635366667 container died 402a02de89575c0cfe2f15b6c5ebc43758807bccc94b412f3c7a4963f6dd2233 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-71c5cfc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T00:57:26.254 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jun 01 00:57:26 smithi079 podman[82322]: 2024-06-01 00:57:26.232991354 +0000 UTC m=+0.521424673 container remove 402a02de89575c0cfe2f15b6c5ebc43758807bccc94b412f3c7a4963f6dd2233 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, RELEASE=reef-71c5cfc, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-01T00:57:26.260 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-01T00:57:26.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:26.299 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-01T00:57:26.299 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-01T00:57:26.300 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.3 2024-06-01T00:57:26.599 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:26 smithi160 systemd[1]: Stopping Ceph osd.3 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:57:27.054 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:26 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3[42480]: 2024-06-01T00:57:26.598+0000 7f21b2866640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:27.054 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:26 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3[42480]: 2024-06-01T00:57:26.599+0000 7f21b2866640 -1 osd.3 41 *** Got signal Terminated *** 2024-06-01T00:57:27.054 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:26 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3[42480]: 2024-06-01T00:57:26.599+0000 7f21b2866640 -1 osd.3 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T00:57:29.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:29 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:29.409+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:30.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:30 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:30.414+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:31.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:31 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:31.393+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:32.236 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:31 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:31.915+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:32.237 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:31 smithi160 podman[60873]: 2024-06-01 00:57:31.958535218 +0000 UTC m=+5.460116069 container died b7c1166cc765e8cef728dbe075fb5dfb379064c47b0dea1f88bb27050946d4de (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3, RELEASE=reef-71c5cfc, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b) 2024-06-01T00:57:32.554 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:32 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:32.398+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:32.554 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:32 smithi160 podman[60873]: 2024-06-01 00:57:32.332574238 +0000 UTC m=+5.834155088 container cleanup b7c1166cc765e8cef728dbe075fb5dfb379064c47b0dea1f88bb27050946d4de (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:32.555 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:32 smithi160 bash[60873]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3 2024-06-01T00:57:32.896 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:32 smithi160 podman[60888]: 2024-06-01 00:57:32.633232739 +0000 UTC m=+0.669381530 container remove b7c1166cc765e8cef728dbe075fb5dfb379064c47b0dea1f88bb27050946d4de (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=reef-71c5cfc, org.label-schema.build-date=20240528) 2024-06-01T00:57:33.236 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:32 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:32.895+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:33.236 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:33 smithi160 podman[60979]: 2024-06-01 00:57:32.902971923 +0000 UTC m=+0.017114689 image pull 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58 2024-06-01T00:57:33.236 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:33 smithi160 podman[60979]: 2024-06-01 00:57:33.009115577 +0000 UTC m=+0.123258374 container create 887d52ccb4d50d797b5cf75201f923f45295b5f34947b3a0b31bb777372b8a5f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, ceph=True, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-01T00:57:33.237 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:33 smithi160 podman[60979]: 2024-06-01 00:57:33.236541705 +0000 UTC m=+0.350684473 container init 887d52ccb4d50d797b5cf75201f923f45295b5f34947b3a0b31bb777372b8a5f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T00:57:33.501 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:33 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:33.366+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:33.501 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:33 smithi160 podman[60979]: 2024-06-01 00:57:33.240107998 +0000 UTC m=+0.354250867 container start 887d52ccb4d50d797b5cf75201f923f45295b5f34947b3a0b31bb777372b8a5f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=reef-71c5cfc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.license=GPLv2) 2024-06-01T00:57:33.501 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:33 smithi160 podman[60979]: 2024-06-01 00:57:33.353226783 +0000 UTC m=+0.467369545 container attach 887d52ccb4d50d797b5cf75201f923f45295b5f34947b3a0b31bb777372b8a5f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3-deactivate, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, GIT_BRANCH=HEAD, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-01T00:57:33.789 INFO:journalctl@ceph.osd.3.smithi160.stdout:Jun 01 00:57:33 smithi160 podman[60979]: 2024-06-01 00:57:33.501460305 +0000 UTC m=+0.615603081 container died 887d52ccb4d50d797b5cf75201f923f45295b5f34947b3a0b31bb777372b8a5f (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=reef-71c5cfc, CEPH_POINT_RELEASE=, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b) 2024-06-01T00:57:34.054 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:33 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:33.859+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:34.087 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-01T00:57:34.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:34.130 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-01T00:57:34.130 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-01T00:57:34.130 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.4 2024-06-01T00:57:34.353 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:34 smithi160 systemd[1]: Stopping Ceph osd.4 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:57:34.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:34 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:34.379+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:34.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:34 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:34.470+0000 7feeae562640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:34.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:34 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:34.470+0000 7feeae562640 -1 osd.4 41 *** Got signal Terminated *** 2024-06-01T00:57:34.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:34 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:34.470+0000 7feeae562640 -1 osd.4 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T00:57:35.289 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:34 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:34.820+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:35.554 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:35 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:35.417+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:36.054 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:35 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:35.792+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:36.775 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:36 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:36.384+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:37.054 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:36 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:36.774+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:37.791 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:37 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:37.413+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:38.054 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:37 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:37.790+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:38.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:38 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:38.400+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:39.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:38 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:38.840+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:39.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:39 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4[48888]: 2024-06-01T00:57:39.445+0000 7feeab366640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:06.895204+0000 front 2024-06-01T00:57:06.895213+0000 (oldest deadline 2024-06-01T00:57:29.195065+0000) 2024-06-01T00:57:40.262 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:39 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:39.849+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:40.262 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:39 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:39.849+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:40.263 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:39 smithi160 podman[61085]: 2024-06-01 00:57:39.969172287 +0000 UTC m=+5.610866187 container stop 9faaf93697e30654b853d898695c8f2be869cc81dde1edad6993e05ccb6fc6ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, ceph=True, RELEASE=reef-71c5cfc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-01T00:57:40.263 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:39 smithi160 podman[61085]: 2024-06-01 00:57:39.995234005 +0000 UTC m=+5.636927901 container died 9faaf93697e30654b853d898695c8f2be869cc81dde1edad6993e05ccb6fc6ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-01T00:57:40.521 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:40 smithi160 podman[61085]: 2024-06-01 00:57:40.353473823 +0000 UTC m=+5.995167710 container cleanup 9faaf93697e30654b853d898695c8f2be869cc81dde1edad6993e05ccb6fc6ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-71c5cfc, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-06-01T00:57:40.521 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:40 smithi160 bash[61085]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4 2024-06-01T00:57:40.804 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:40 smithi160 podman[61111]: 2024-06-01 00:57:40.52148308 +0000 UTC m=+0.544249795 container remove 9faaf93697e30654b853d898695c8f2be869cc81dde1edad6993e05ccb6fc6ba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, ceph=True) 2024-06-01T00:57:41.132 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:40 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:40.830+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:41.132 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:40 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:40.830+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:41.133 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:40 smithi160 podman[61206]: 2024-06-01 00:57:40.922124233 +0000 UTC m=+0.103447997 container create a3ebbf84fbe497832a816947d7628b5e2cb1d0a911dc446c46b9a3a2337d7fba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.build-date=20240528, RELEASE=reef-71c5cfc, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.35.3, GIT_CLEAN=True) 2024-06-01T00:57:41.133 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:40 smithi160 podman[61206]: 2024-06-01 00:57:40.836452893 +0000 UTC m=+0.017776661 image pull 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58 2024-06-01T00:57:41.133 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:41 smithi160 podman[61206]: 2024-06-01 00:57:41.132920936 +0000 UTC m=+0.314244700 container init a3ebbf84fbe497832a816947d7628b5e2cb1d0a911dc446c46b9a3a2337d7fba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-71c5cfc, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, io.buildah.version=1.35.3) 2024-06-01T00:57:41.391 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:41 smithi160 podman[61206]: 2024-06-01 00:57:41.1362827 +0000 UTC m=+0.317606463 container start a3ebbf84fbe497832a816947d7628b5e2cb1d0a911dc446c46b9a3a2337d7fba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-71c5cfc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, ceph=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.35.3) 2024-06-01T00:57:41.391 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:41 smithi160 podman[61206]: 2024-06-01 00:57:41.208045559 +0000 UTC m=+0.389369321 container attach a3ebbf84fbe497832a816947d7628b5e2cb1d0a911dc446c46b9a3a2337d7fba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_CLEAN=True, RELEASE=reef-71c5cfc, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:41.677 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:41 smithi160 podman[61206]: 2024-06-01 00:57:41.391586901 +0000 UTC m=+0.572910672 container died a3ebbf84fbe497832a816947d7628b5e2cb1d0a911dc446c46b9a3a2337d7fba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_CLEAN=True, RELEASE=reef-71c5cfc, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-06-01T00:57:41.938 INFO:journalctl@ceph.osd.4.smithi160.stdout:Jun 01 00:57:41 smithi160 podman[61228]: 2024-06-01 00:57:41.919680257 +0000 UTC m=+0.522488160 container remove a3ebbf84fbe497832a816947d7628b5e2cb1d0a911dc446c46b9a3a2337d7fba (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.35.3, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-71c5cfc, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-01T00:57:41.938 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:41 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:41.880+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:41.938 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:41 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:41.880+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:41.943 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-01T00:57:41.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:41.983 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-01T00:57:41.983 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-01T00:57:41.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.5 2024-06-01T00:57:42.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:42 smithi160 systemd[1]: Stopping Ceph osd.5 for 22870f5e-1fb0-11ef-bc9b-c7b262605968... 2024-06-01T00:57:42.804 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:42 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:42.322+0000 7f6673929640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T00:57:42.804 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:42 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:42.322+0000 7f6673929640 -1 osd.5 41 *** Got signal Terminated *** 2024-06-01T00:57:42.804 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:42 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:42.322+0000 7f6673929640 -1 osd.5 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T00:57:43.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:42 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:42.836+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:43.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:42 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:42.836+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:44.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:43 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:43.871+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:44.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:43 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:43.871+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:45.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:44 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:44.879+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:45.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:44 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:44.879+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:46.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:45 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:45.913+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:46.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:45 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:45.913+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:46.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:45 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:45.913+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-06-01T00:57:23.412929+0000 front 2024-06-01T00:57:23.413006+0000 (oldest deadline 2024-06-01T00:57:45.112850+0000) 2024-06-01T00:57:47.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:46 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:46.901+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-06-01T00:57:05.811723+0000 front 2024-06-01T00:57:05.811861+0000 (oldest deadline 2024-06-01T00:57:31.711584+0000) 2024-06-01T00:57:47.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:46 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:46.901+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-06-01T00:57:15.212343+0000 front 2024-06-01T00:57:15.212356+0000 (oldest deadline 2024-06-01T00:57:39.312197+0000) 2024-06-01T00:57:47.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:46 smithi160 ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5[55814]: 2024-06-01T00:57:46.901+0000 7f666c725640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-06-01T00:57:23.412929+0000 front 2024-06-01T00:57:23.413006+0000 (oldest deadline 2024-06-01T00:57:45.112850+0000) 2024-06-01T00:57:47.998 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:47 smithi160 podman[61306]: 2024-06-01 00:57:47.727937753 +0000 UTC m=+5.510874001 container died f38b70b5cff4f66726afed816799ca30ba6d84141c63b8b103b3a8b744321c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, RELEASE=reef-71c5cfc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-01T00:57:48.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:48 smithi160 podman[61306]: 2024-06-01 00:57:48.097128792 +0000 UTC m=+5.880065034 container cleanup f38b70b5cff4f66726afed816799ca30ba6d84141c63b8b103b3a8b744321c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=reef-71c5cfc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:48.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:48 smithi160 bash[61306]: ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5 2024-06-01T00:57:48.714 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:48 smithi160 podman[61323]: 2024-06-01 00:57:48.35632746 +0000 UTC m=+0.620571791 container remove f38b70b5cff4f66726afed816799ca30ba6d84141c63b8b103b3a8b744321c62 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5, org.label-schema.build-date=20240528, RELEASE=reef-71c5cfc, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-01T00:57:49.001 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:48 smithi160 podman[61420]: 2024-06-01 00:57:48.61717706 +0000 UTC m=+0.017480865 image pull 53681e237463a12768647ecd88d52a5f5bb8401870e6f50a66d67382d8b64ffe quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58 2024-06-01T00:57:49.001 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:48 smithi160 podman[61420]: 2024-06-01 00:57:48.724002341 +0000 UTC m=+0.124306159 container create 7bcfe12150bf4aebbe9cc9f4271d17c48d39a56eb7d51b397a79c6e2c120d044 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=reef-71c5cfc) 2024-06-01T00:57:49.001 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:48 smithi160 podman[61420]: 2024-06-01 00:57:48.926489369 +0000 UTC m=+0.326793173 container init 7bcfe12150bf4aebbe9cc9f4271d17c48d39a56eb7d51b397a79c6e2c120d044 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, RELEASE=reef-71c5cfc, org.label-schema.build-date=20240528) 2024-06-01T00:57:49.001 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:48 smithi160 podman[61420]: 2024-06-01 00:57:48.93012801 +0000 UTC m=+0.330431808 container start 7bcfe12150bf4aebbe9cc9f4271d17c48d39a56eb7d51b397a79c6e2c120d044 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.vendor=CentOS, RELEASE=reef-71c5cfc, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3) 2024-06-01T00:57:49.001 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:49 smithi160 podman[61420]: 2024-06-01 00:57:49.001039114 +0000 UTC m=+0.401342915 container attach 7bcfe12150bf4aebbe9cc9f4271d17c48d39a56eb7d51b397a79c6e2c120d044 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5-deactivate, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-06-01T00:57:49.304 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:49 smithi160 podman[61420]: 2024-06-01 00:57:49.182845881 +0000 UTC m=+0.583149677 container died 7bcfe12150bf4aebbe9cc9f4271d17c48d39a56eb7d51b397a79c6e2c120d044 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-71c5cfc, GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True) 2024-06-01T00:57:49.713 INFO:journalctl@ceph.osd.5.smithi160.stdout:Jun 01 00:57:49 smithi160 podman[61444]: 2024-06-01 00:57:49.695699582 +0000 UTC m=+0.506866361 container remove 7bcfe12150bf4aebbe9cc9f4271d17c48d39a56eb7d51b397a79c6e2c120d044 (image=quay.ceph.io/ceph-ci/ceph@sha256:7f3f24c471a4b43e7555a75386543446a04ec865e893588e6a5d3cc2941c8c58, name=ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=985bb830dd12db4967d238deb1cf64ef5cdf187b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=reef-71c5cfc, maintainer=Guillaume Abrioux ) 2024-06-01T00:57:49.719 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-22870f5e-1fb0-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-01T00:57:49.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T00:57:49.757 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-01T00:57:49.758 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-01T00:57:49.915 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:58:00.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-01T00:58:00.213 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:58:05.969 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T00:58:06.003 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T00:58:06.030 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-01T00:58:06.032 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360/remote/smithi079/crash 2024-06-01T00:58:06.033 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/crash -- . 2024-06-01T00:58:06.092 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-01T00:58:06.092 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now 2024-06-01T00:58:06.094 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360/remote/smithi160/crash 2024-06-01T00:58:06.097 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/crash -- . 2024-06-01T00:58:06.165 INFO:teuthology.orchestra.run.smithi160.stderr:tar: /var/lib/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-01T00:58:06.165 INFO:teuthology.orchestra.run.smithi160.stderr:tar: Error is not recoverable: exiting now 2024-06-01T00:58:06.167 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-01T00:58:06.167 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-01T00:58:06.203 INFO:teuthology.orchestra.run.smithi079.stdout:2024-06-01T00:46:49.015545+0000 mon.smithi079 (mon.0) 120 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-01T00:58:06.204 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-01T00:58:06.204 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[SEC\]' /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-01T00:58:06.278 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]' /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-01T00:58:06.350 INFO:teuthology.orchestra.run.smithi079.stdout:2024-06-01T00:48:03.118593+0000 mgr.smithi160.dxcuke (mgr.14238) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents 2024-06-01T00:58:06.351 INFO:tasks.cephadm:Compressing logs... 2024-06-01T00:58:06.352 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-01T00:58:06.394 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-01T00:58:06.417 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-01T00:58:06.417 INFO:teuthology.orchestra.run.smithi079.stderr:: No such file or directory 2024-06-01T00:58:06.417 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log 2024-06-01T00:58:06.417 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-01T00:58:06.417 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.audit.log: /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mon.smithi079.log 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 88.7% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mon.smithi079.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi079.log 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mgr.smithi079.emylav.log 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr: 84.8% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-01T00:58:06.422 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi079.log: 90.8% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-01T00:58:06.423 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:: No such file or directory 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.audit.log: 92.2%gzip -- replaced with /var/log/ceph/cephadm.log.gz -5 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr: 88.9% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-01T00:58:06.424 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mon.smithi160.log 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr: 91.0% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mon.smithi160.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mgr.smithi160.dxcuke.log 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr: 83.6% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi160.log 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mgr.smithi160.dxcuke.log: /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-volume.log: /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi160.log: 93.6% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi160.log.gz 2024-06-01T00:58:06.425 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mgr.smithi079.emylav.log: 93.6% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-client.ceph-exporter.smithi079.log.gz 2024-06-01T00:58:06.427 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mgr.smithi160.dxcuke.log.gz 2024-06-01T00:58:06.428 INFO:teuthology.orchestra.run.smithi160.stderr: 90.9% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-01T00:58:06.429 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-01T00:58:06.432 INFO:teuthology.orchestra.run.smithi079.stderr: 91.0% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-01T00:58:06.464 INFO:teuthology.orchestra.run.smithi160.stderr: 93.0% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-01T00:58:06.472 INFO:teuthology.orchestra.run.smithi079.stderr: 93.1% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-01T00:58:06.474 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-01T00:58:06.475 INFO:teuthology.orchestra.run.smithi160.stderr: 93.0% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-01T00:58:06.504 INFO:teuthology.orchestra.run.smithi079.stderr: 92.8% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-01T00:58:06.516 INFO:teuthology.orchestra.run.smithi160.stderr: 92.9% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mon.smithi160.log.gz 2024-06-01T00:58:06.518 INFO:teuthology.orchestra.run.smithi160.stderr: 2024-06-01T00:58:06.518 INFO:teuthology.orchestra.run.smithi160.stderr:real 0m0.109s 2024-06-01T00:58:06.518 INFO:teuthology.orchestra.run.smithi160.stderr:user 0m0.265s 2024-06-01T00:58:06.518 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m0.030s 2024-06-01T00:58:06.526 INFO:teuthology.orchestra.run.smithi079.stderr: 89.0% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mgr.smithi079.emylav.log.gz 2024-06-01T00:58:06.530 INFO:teuthology.orchestra.run.smithi079.stderr: 93.3% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-01T00:58:06.694 INFO:teuthology.orchestra.run.smithi079.stderr: 90.5% -- replaced with /var/log/ceph/22870f5e-1fb0-11ef-bc9b-c7b262605968/ceph-mon.smithi079.log.gz 2024-06-01T00:58:06.697 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-06-01T00:58:06.697 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m0.290s 2024-06-01T00:58:06.697 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m0.641s 2024-06-01T00:58:06.697 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.049s 2024-06-01T00:58:06.697 INFO:tasks.cephadm:Archiving logs... 2024-06-01T00:58:06.697 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360/remote/smithi079/log 2024-06-01T00:58:06.698 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-01T00:58:06.869 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360/remote/smithi160/log 2024-06-01T00:58:06.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-01T00:58:06.941 INFO:tasks.cephadm:Removing cluster... 2024-06-01T00:58:06.941 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 --force 2024-06-01T00:58:07.083 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:58:07.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 22870f5e-1fb0-11ef-bc9b-c7b262605968 --force 2024-06-01T00:58:07.490 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 22870f5e-1fb0-11ef-bc9b-c7b262605968 2024-06-01T00:58:07.756 INFO:tasks.cephadm:Removing cephadm ... 2024-06-01T00:58:07.757 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-01T00:58:07.772 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-01T00:58:07.788 INFO:tasks.cephadm:Teardown complete 2024-06-01T00:58:07.788 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_036ad575e36eb2f5d36bafd4f61b4922eadba901/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-01T00:58:07.951 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8819141184be4a0daf0530a23b9730c1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_036ad575e36eb2f5d36bafd4f61b4922eadba901/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f1316ee19712baed461a5bda1a1cefe32b1d255a/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-01T00:58:07.953 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-01T00:58:08.009 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-01T00:58:08.017 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-01T00:58:08.017 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-01T00:58:08.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-01T00:58:08.106 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-01T00:58:08.115 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-01T00:58:08.115 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T00:58:08.117 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T00:58:08.129 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-06-01T00:58:08.131 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-06-01T00:58:08.134 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T00:58:08.134 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-06-01T00:58:08.134 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +809us[ +813us] +/- 50ms 2024-06-01T00:58:08.135 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -750us[ -745us] +/- 68ms 2024-06-01T00:58:08.135 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 -448us[ -448us] +/- 89ms 2024-06-01T00:58:08.135 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:58:08.136 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T00:58:08.137 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-06-01T00:58:08.137 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv01.front.sepia.ceph.com 3 6 377 47 +820us[ +819us] +/- 50ms 2024-06-01T00:58:08.137 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 42 -718us[ -718us] +/- 68ms 2024-06-01T00:58:08.137 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 -434us[ -434us] +/- 89ms 2024-06-01T00:58:08.137 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T00:58:08.137 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-01T00:58:08.146 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-01T00:58:08.146 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-01T00:58:08.190 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-01T00:58:08.260 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-01T00:58:08.325 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-01T00:58:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:58:08.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-01T00:58:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:58:08.380 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-01T00:58:08.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-01T00:58:08.397 INFO:teuthology.task.internal:Duration was 1228.245462 seconds 2024-06-01T00:58:08.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-01T00:58:08.406 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-01T00:58:08.406 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-01T00:58:08.409 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-01T00:58:08.444 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T00:58:08.458 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T00:58:08.836 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-01T00:58:08.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-06-01T00:58:08.837 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-01T00:58:08.857 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2024-06-01T00:58:08.857 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-01T00:58:08.897 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-01T00:58:08.897 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-01T00:58:08.900 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-01T00:58:08.964 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-01T00:58:08.964 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-01T00:58:08.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-01T00:58:09.342 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-01T00:58:09.355 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-01T00:58:09.355 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-01T00:58:09.380 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-01T00:58:09.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-01T00:58:09.417 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-01T00:58:09.422 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-01T00:58:09.443 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-06-01T00:58:09.469 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2024-06-01T00:58:09.491 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-01T00:58:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:58:09.516 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-01T00:58:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T00:58:09.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-01T00:58:09.562 INFO:teuthology.task.internal:Transferring archived files... 2024-06-01T00:58:09.563 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360/remote/smithi079 2024-06-01T00:58:09.563 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-01T00:58:09.604 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-31_02:08:10-upgrade-squid-distro-default-smithi/7735360/remote/smithi160 2024-06-01T00:58:09.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-01T00:58:09.639 INFO:teuthology.task.internal:Removing archive directory... 2024-06-01T00:58:09.640 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-01T00:58:09.642 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-01T00:58:09.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-01T00:58:09.701 INFO:teuthology.task.internal:Not uploading archives. 2024-06-01T00:58:09.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-01T00:58:09.711 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-01T00:58:09.711 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-01T00:58:09.714 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-01T00:58:09.726 INFO:teuthology.orchestra.run.smithi079.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 1 00:58 /home/ubuntu/cephtest 2024-06-01T00:58:09.744 INFO:teuthology.orchestra.run.smithi160.stdout: 265316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 1 00:58 /home/ubuntu/cephtest 2024-06-01T00:58:09.745 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-01T00:58:09.755 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-01T00:58:09.783 INFO:teuthology.run:Summary data: description: upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1228.2454617023468 failure_reason: '"2024-06-01T00:48:03.118593+0000 mgr.smithi160.dxcuke (mgr.14238) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8819141184be4a0daf0530a23b9730c1 status: fail success: false 2024-06-01T00:58:09.783 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-01T00:58:09.871 INFO:teuthology.run:FAIL