2024-01-19T01:03:14.656 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-19T01:03:14.657 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7520817 --verbose --owner scheduled_yuriw@teuthology --name yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7520817/orig.config.yaml 2024-01-19T01:03:14.710 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-19T01:03:14.798 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7520817 branch: wip-yuri8-testing-2024-01-18-0823 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7520817' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2024-01-18-0823 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0edf41a622739843d6b978b179ff3227b476dd9d ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0edf41a622739843d6b978b179ff3227b476dd9d kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2024-01-18-0823 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6631 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2024-01-18-0823 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0edf41a622739843d6b978b179ff3227b476dd9d targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOb6h1mRBpZAS+08M5EdF2qZcKUF+lcBwYItW8/nYGc4IqW9Cp8E8K99obhX3W/PzVFh0QI7SxDBbSbJaIv6KYs= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8pWPkJbX6ZdmNwxl8Dq2TIj9AcPRpZj+vWnoNt3dAcMQHnU1SxBsZMgT8K2fuqE3yJGb2FQiAT6gdteIbXqCQ= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/yuriw/src/github.com_ceph_ceph-c_wip-yuri8-testing-2024-01-18-0823/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-18_21:19:14 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-19T01:03:14.799 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa; will attempt to use it 2024-01-19T01:03:14.799 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks 2024-01-19T01:03:14.800 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-19T01:03:14.801 INFO:teuthology.task.internal:Checking packages... 2024-01-19T01:03:14.828 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0edf41a622739843d6b978b179ff3227b476dd9d' 2024-01-19T01:03:14.828 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T01:03:14.828 INFO:teuthology.packaging:ref: None 2024-01-19T01:03:14.828 INFO:teuthology.packaging:tag: None 2024-01-19T01:03:14.828 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-01-18-0823 2024-01-19T01:03:14.828 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:03:14.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2024-01-18-0823 2024-01-19T01:03:15.054 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-766.g0edf41a6 2024-01-19T01:03:15.055 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-19T01:03:15.062 INFO:teuthology.task.internal:no buildpackages task found 2024-01-19T01:03:15.062 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-19T01:03:15.070 INFO:teuthology.task.internal:Saving configuration 2024-01-19T01:03:15.089 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-19T01:03:15.094 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-19T01:03:15.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7520817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 00:57:51.247720', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOb6h1mRBpZAS+08M5EdF2qZcKUF+lcBwYItW8/nYGc4IqW9Cp8E8K99obhX3W/PzVFh0QI7SxDBbSbJaIv6KYs='} 2024-01-19T01:03:15.144 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-18_21:19:14-fs-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7520817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 00:57:51.249198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD8pWPkJbX6ZdmNwxl8Dq2TIj9AcPRpZj+vWnoNt3dAcMQHnU1SxBsZMgT8K2fuqE3yJGb2FQiAT6gdteIbXqCQ='} 2024-01-19T01:03:15.144 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-19T01:03:15.151 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-19T01:03:15.151 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-19T01:03:15.151 INFO:teuthology.run_tasks:Running task console_log... 2024-01-19T01:03:15.225 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f903418c160>, signals=[15]) 2024-01-19T01:03:15.225 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-19T01:03:15.232 INFO:teuthology.task.internal:Opening connections... 2024-01-19T01:03:15.232 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:03:15.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:03:15.327 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-01-19T01:03:15.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:03:15.428 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-19T01:03:15.530 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-01-19T01:03:15.549 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-01-19T01:03:15.550 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-01-19T01:03:15.607 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-19T01:03:15.608 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-19T01:03:15.609 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-01-19T01:03:15.635 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-01-19T01:03:15.654 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-01-19T01:03:15.655 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-01-19T01:03:15.724 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-01-19T01:03:15.724 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-01-19T01:03:15.724 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-01-19T01:03:15.724 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-01-19T01:03:15.724 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-01-19T01:03:15.724 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-01-19T01:03:15.724 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-19T01:03:15.725 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-01-19T01:03:15.725 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-01-19T01:03:15.725 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-19T01:03:15.725 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-01-19T01:03:15.725 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-19T01:03:15.725 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-19T01:03:15.725 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-19T01:03:15.725 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-01-19T01:03:15.759 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-19T01:03:15.792 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-19T01:03:15.834 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-19T01:03:15.834 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-01-19T01:03:15.840 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-01-19T01:03:15.857 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-19T01:03:15.871 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-19T01:03:15.871 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-01-19T01:03:15.899 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-01-19T01:03:16.018 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-19T01:03:16.220 INFO:teuthology.run_tasks:Running task kernel... 2024-01-19T01:03:16.298 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-19T01:03:16.299 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-19T01:03:16.299 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-19T01:03:16.300 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-19T01:03:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T01:03:16.317 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-01-19T01:03:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T01:03:16.318 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-01-19T01:03:16.374 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-361.el9.x86_64 2024-01-19T01:03:16.375 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-361.el9.x86_64 2024-01-19T01:03:16.375 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-01-19T01:03:16.380 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-361.el9.x86_64 2024-01-19T01:03:16.380 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-361.el9.x86_64 2024-01-19T01:03:16.380 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-01-19T01:03:18.675 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2024-01-19T01:03:18.732 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2024-01-19T01:03:20.250 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-01-19T01:03:20.366 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-01-19T01:03:21.324 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-01-19T01:03:22.831 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-01-19T01:03:24.849 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-01-19T01:03:25.574 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-01-19T01:03:30.475 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.9 MB 00:01 2024-01-19T01:03:30.483 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.9 MB 00:02 2024-01-19T01:03:31.934 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2024-01-19T01:03:32.017 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-01-19T01:03:33.379 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-01-19T01:03:33.429 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-01-19T01:03:38.227 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-01-19T01:03:38.534 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-01-19T01:03:40.434 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-19T01:03:40.434 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-19T01:03:40.581 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:03:40.593 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:03:40.593 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:03:40.593 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:03:40.593 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:03:40.593 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-408.el9 CentOS-BaseOS 6.0 M 2024-01-19T01:03:40.593 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:03:40.593 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-408.el9 CentOS-BaseOS 21 M 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-408.el9 CentOS-BaseOS 39 M 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-408.el9 CentOS-BaseOS 33 M 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 99 M 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 122 M 2024-01-19T01:03:40.594 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:03:40.741 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-19T01:03:40.741 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-19T01:03:40.863 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-408.el9 CentOS-BaseOS 6.0 M 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-408.el9 CentOS-BaseOS 21 M 2024-01-19T01:03:40.872 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-408.el9 CentOS-BaseOS 39 M 2024-01-19T01:03:40.873 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-408.el9 CentOS-BaseOS 33 M 2024-01-19T01:03:40.873 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:03:40.873 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:03:40.873 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:03:40.873 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-01-19T01:03:40.873 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:03:40.874 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 99 M 2024-01-19T01:03:40.874 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 122 M 2024-01-19T01:03:40.874 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:03:41.095 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-408.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-01-19T01:03:41.635 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-408.el9.x86_64.rpm 7.9 MB/s | 6.0 MB 00:00 2024-01-19T01:03:41.962 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-408.el9.x86_64.rpm 15 MB/s | 21 MB 00:01 2024-01-19T01:03:42.743 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-core-5.14.0-408.el9.x86_64.rpm 11 MB/s | 21 MB 00:01 2024-01-19T01:03:43.520 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-core-5.14.0-408.el9.x86_6 14 MB/s | 33 MB 00:02 2024-01-19T01:03:43.677 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-modules-core-5.14.0-408.el9.x86_6 16 MB/s | 33 MB 00:02 2024-01-19T01:03:44.027 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-5.14.0-408.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-01-19T01:03:44.030 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:03:44.030 INFO:teuthology.orchestra.run.smithi079.stdout:Total 31 MB/s | 99 MB 00:03 2024-01-19T01:03:44.031 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:03:44.120 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-5.14.0-408.el9.x86_64.rpm 11 MB/s | 39 MB 00:03 2024-01-19T01:03:44.124 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:03:44.124 INFO:teuthology.orchestra.run.smithi152.stdout:Total 28 MB/s | 99 MB 00:03 2024-01-19T01:03:44.124 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:03:44.401 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:03:44.401 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:03:44.490 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:03:44.490 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:03:45.453 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:03:45.453 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:03:45.508 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:03:45.508 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:03:47.851 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:03:47.904 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:03:48.510 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-408.el9.x86_64 1/4 2024-01-19T01:03:48.567 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-408.el9.x86_64 1/4 2024-01-19T01:03:48.643 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T01:03:48.691 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T01:03:51.374 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T01:03:51.474 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T01:03:51.561 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T01:03:51.654 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T01:03:56.250 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T01:03:56.332 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T01:03:56.364 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:03:56.584 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:01.091 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:01.292 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:30.463 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:30.464 INFO:teuthology.orchestra.run.smithi079.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-19T01:04:30.464 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:04:30.533 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:30.602 INFO:teuthology.orchestra.run.smithi152.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-19T01:04:30.603 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:04:30.673 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:30.971 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:30.971 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-408.el9.x86_64 1/4 2024-01-19T01:04:30.971 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T01:04:30.971 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T01:04:31.141 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:31.141 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-408.el9.x86_64 1/4 2024-01-19T01:04:31.141 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-19T01:04:31.141 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-19T01:04:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:31.437 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:04:31.438 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:04:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.438 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:04:31.438 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-408.el9.x86_64 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:04:31.589 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:04:31.936 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T01:04:32.070 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T01:04:32.070 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.071 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.071 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-19T01:04:32.071 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-19T01:04:32.071 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-19T01:04:32.071 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-01-19T01:04:32.072 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T01:04:32.105 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T01:04:32.178 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T01:04:32.178 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.179 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-01-19T01:04:32.212 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-01-19T01:04:32.212 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-01-19T01:04:32.212 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-01-19T01:04:32.212 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 68.el9 2024-01-19T01:04:32.212 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-01-19T01:04:32.212 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-19T01:04:32.212 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8264254 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-19T01:04:32.213 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-19T01:04:32.214 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-01-19T01:04:32.214 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:04:32.214 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-01-19T01:04:32.214 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-19T01:04:32.214 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.215 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-19T01:04:32.270 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T01:04:32.270 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.270 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.270 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-19T01:04:32.270 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-19T01:04:32.270 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-19T01:04:32.270 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-01-19T01:04:32.270 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T01:04:32.377 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-19T01:04:32.378 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.378 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 68.el9 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8264254 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-19T01:04:32.419 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:04:32.420 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-01-19T01:04:32.421 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-19T01:04:32.421 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-408.el9.x86_64 2024-01-19T01:04:32.421 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-19T01:04:32.885 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-01-19T01:04:33.048 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-01-19T01:04:34.193 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-19T01:04:34.236 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-01-19T01:04:34.239 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-01-19T01:04:34.254 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.OYQvm7kixM 2024-01-19T01:04:34.255 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OYQvm7kixM 2024-01-19T01:04:34.326 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.OYQvm7kixM 2024-01-19T01:04:34.463 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.OYQvm7kixM is 6KB 2024-01-19T01:04:34.467 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-19T01:04:34.498 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-01-19T01:04:34.500 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-01-19T01:04:34.511 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.OYQvm7kixM 2024-01-19T01:04:34.515 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.5RsR4r2EYx 2024-01-19T01:04:34.516 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5RsR4r2EYx 2024-01-19T01:04:34.525 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-01-19T01:04:34.584 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.5RsR4r2EYx 2024-01-19T01:04:34.605 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-19T01:04:34.605 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-19T01:04:34.605 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-19T01:04:34.606 INFO:teuthology.orchestra.run.smithi079.stdout:15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64.conf 2024-01-19T01:04:34.607 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64 2024-01-19T01:04:34.726 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.5RsR4r2EYx is 6KB 2024-01-19T01:04:34.772 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-01-19T01:04:34.776 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.5RsR4r2EYx 2024-01-19T01:04:34.789 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-01-19T01:04:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-19T01:04:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-19T01:04:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-19T01:04:34.864 INFO:teuthology.orchestra.run.smithi152.stdout:15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64.conf 2024-01-19T01:04:34.866 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64 2024-01-19T01:04:35.023 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-01-19T01:05:04.806 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-19T01:05:04.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-01-19T01:05:04.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:05:05.026 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-19T01:05:05.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-01-19T01:05:05.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:05:23.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-01-19T01:05:23.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-01-19T01:05:32.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-01-19T01:05:32.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:05:32.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-01-19T01:05:32.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:05:35.324 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-01-19T01:05:35.580 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-01-19T01:05:47.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-01-19T01:05:47.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:05:47.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-01-19T01:05:47.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:06:05.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-01-19T01:06:20.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-01-19T01:06:20.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:06:21.113 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-01-19T01:06:21.657 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-01-19T01:06:21.658 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-408.el9.x86_64"... 2024-01-19T01:06:21.658 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-01-19T01:06:21.683 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-408.el9.x86_64 2024-01-19T01:06:21.684 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-408.el9.x86_64 vs 5.14.0-408.el9.x86_64 2024-01-19T01:06:21.684 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-19T01:06:21.684 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-19T01:06:22.684 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-19T01:06:22.685 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-19T01:06:22.759 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-01-19T01:06:22.934 DEBUG:teuthology.parallel:result is None 2024-01-19T01:06:47.583 DEBUG:teuthology.orchestra.remote:timed out 2024-01-19T01:07:02.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-01-19T01:07:02.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:07:02.855 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-01-19T01:07:03.243 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-01-19T01:07:03.243 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-408.el9.x86_64"... 2024-01-19T01:07:03.243 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-01-19T01:07:03.257 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-408.el9.x86_64 2024-01-19T01:07:03.258 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-408.el9.x86_64 vs 5.14.0-408.el9.x86_64 2024-01-19T01:07:03.258 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-19T01:07:03.258 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-19T01:07:04.259 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-19T01:07:04.259 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-19T01:07:04.338 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-01-19T01:07:04.383 DEBUG:teuthology.parallel:result is None 2024-01-19T01:07:04.383 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-19T01:07:04.392 INFO:teuthology.task.internal:Creating test directory... 2024-01-19T01:07:04.392 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-19T01:07:04.395 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-19T01:07:04.412 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-19T01:07:04.418 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-19T01:07:04.428 INFO:teuthology.task.internal:Creating archive directory... 2024-01-19T01:07:04.428 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-19T01:07:04.453 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-19T01:07:04.501 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-19T01:07:04.507 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-19T01:07:04.507 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-19T01:07:04.510 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-19T01:07:04.546 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T01:07:04.552 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T01:07:04.559 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T01:07:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T01:07:04.568 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-19T01:07:04.574 INFO:teuthology.task.internal:Configuring sudo... 2024-01-19T01:07:04.574 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-19T01:07:04.604 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-19T01:07:04.637 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-19T01:07:04.646 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-19T01:07:04.646 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-19T01:07:04.673 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-19T01:07:04.693 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T01:07:04.765 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T01:07:04.822 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T01:07:04.900 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T01:07:04.947 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:07:04.947 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-19T01:07:05.013 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T01:07:05.055 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T01:07:05.115 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T01:07:05.195 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T01:07:05.243 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:07:05.243 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-19T01:07:05.312 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-01-19T01:07:05.314 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-01-19T01:07:05.359 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T01:07:05.390 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T01:07:05.782 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-19T01:07:05.790 INFO:teuthology.task.internal:Starting timer... 2024-01-19T01:07:05.791 INFO:teuthology.run_tasks:Running task pcp... 2024-01-19T01:07:05.848 INFO:teuthology.run_tasks:Running task selinux... 2024-01-19T01:07:05.856 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-19T01:07:05.857 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-01-19T01:07:05.920 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-01-19T01:07:05.922 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-01-19T01:07:05.988 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-01-19T01:07:05.990 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-19T01:07:05.990 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-01-19T01:07:06.014 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-01-19T01:07:06.014 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-01-19T01:07:06.069 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-01-19T01:07:06.069 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-01-19T01:07:06.069 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T01:07:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T01:07:06.096 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T01:07:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T01:07:06.124 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-19T01:07:06.124 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-01-19T01:07:06.173 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-01-19T01:07:06.204 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-19T01:07:06.214 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-19T01:07:06.230 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-19T01:07:06.231 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-19T01:11:05.188 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-01-19T01:11:05.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-01-19T01:11:05.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:11:05.283 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-01-19T01:11:05.367 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-01-19T01:11:05.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-01-19T01:11:05.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T01:11:05.458 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-01-19T01:11:05.534 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-01-19T01:11:05.535 INFO:teuthology.run_tasks:Running task clock... 2024-01-19T01:11:05.546 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-19T01:11:05.546 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-19T01:11:05.547 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T01:11:05.549 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-19T01:11:05.549 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T01:11:05.578 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-19T01:11:05.595 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-19T01:11:05.606 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-19T01:11:05.622 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-19T01:11:05.624 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-01-19T01:11:05.638 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-01-19T01:11:05.651 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-01-19T01:11:05.654 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-19T01:11:05.665 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-01-19T01:11:05.671 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-19T01:11:05.681 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-19T01:11:05.698 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-19T01:11:05.720 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-01-19T01:11:05.724 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T01:11:05.724 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-01-19T01:11:05.724 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.724 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.724 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.724 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.752 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-01-19T01:11:05.756 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T01:11:05.756 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-01-19T01:11:05.756 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.756 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.756 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.756 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T01:11:05.757 INFO:teuthology.run_tasks:Running task install... 2024-01-19T01:11:05.766 DEBUG:teuthology.task.install:project ceph 2024-01-19T01:11:05.766 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}} 2024-01-19T01:11:05.766 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-19T01:11:05.766 INFO:teuthology.task.install:Using flavor: default 2024-01-19T01:11:05.773 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-19T01:11:05.773 INFO:teuthology.task.install:extra packages: [] 2024-01-19T01:11:05.773 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-19T01:11:05.774 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T01:11:05.774 INFO:teuthology.packaging:ref: None 2024-01-19T01:11:05.774 INFO:teuthology.packaging:tag: None 2024-01-19T01:11:05.774 INFO:teuthology.packaging:branch: quincy 2024-01-19T01:11:05.774 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:11:05.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T01:11:05.776 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-19T01:11:05.776 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T01:11:05.777 INFO:teuthology.packaging:ref: None 2024-01-19T01:11:05.777 INFO:teuthology.packaging:tag: None 2024-01-19T01:11:05.777 INFO:teuthology.packaging:branch: quincy 2024-01-19T01:11:05.777 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:11:05.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T01:11:05.960 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/ 2024-01-19T01:11:05.960 INFO:teuthology.task.install.rpm:Package version is 17.2.7-268.g6441305d 2024-01-19T01:11:05.970 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/ 2024-01-19T01:11:05.971 INFO:teuthology.task.install.rpm:Package version is 17.2.7-268.g6441305d 2024-01-19T01:11:06.136 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-19T01:11:06.137 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:11:06.137 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-19T01:11:06.148 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/6441305d08929b8a72c0322a5c268c64b4a99c65/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-19T01:11:06.149 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:11:06.149 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-19T01:11:06.165 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-19T01:11:06.166 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T01:11:06.166 INFO:teuthology.packaging:ref: None 2024-01-19T01:11:06.166 INFO:teuthology.packaging:tag: None 2024-01-19T01:11:06.166 INFO:teuthology.packaging:branch: quincy 2024-01-19T01:11:06.166 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:11:06.166 DEBUG:teuthology.orchestra.run.smithi079:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-19T01:11:06.177 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-19T01:11:06.178 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T01:11:06.178 INFO:teuthology.packaging:ref: None 2024-01-19T01:11:06.178 INFO:teuthology.packaging:tag: None 2024-01-19T01:11:06.178 INFO:teuthology.packaging:branch: quincy 2024-01-19T01:11:06.179 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:11:06.179 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-19T01:11:06.234 DEBUG:teuthology.orchestra.run.smithi079:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-19T01:11:06.245 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-19T01:11:06.313 DEBUG:teuthology.orchestra.run.smithi079:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-19T01:11:06.325 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-19T01:11:06.379 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1 2024-01-19T01:11:06.381 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum clean all 2024-01-19T01:11:06.392 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-01-19T01:11:06.394 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-01-19T01:11:06.620 INFO:teuthology.orchestra.run.smithi079.stdout:73 files removed 2024-01-19T01:11:06.634 INFO:teuthology.orchestra.run.smithi152.stdout:73 files removed 2024-01-19T01:11:06.646 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-radosgw 2024-01-19T01:11:06.661 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-01-19T01:11:07.387 INFO:teuthology.orchestra.run.smithi079.stdout:ceph packages for x86_64 183 kB/s | 77 kB 00:00 2024-01-19T01:11:07.453 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 164 kB/s | 77 kB 00:00 2024-01-19T01:11:07.812 INFO:teuthology.orchestra.run.smithi079.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2024-01-19T01:11:07.853 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 26 kB/s | 9.1 kB 00:00 2024-01-19T01:11:08.147 INFO:teuthology.orchestra.run.smithi079.stdout:ceph source packages 6.1 kB/s | 1.8 kB 00:00 2024-01-19T01:11:08.271 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 4.7 kB/s | 1.8 kB 00:00 2024-01-19T01:11:08.570 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2024-01-19T01:11:08.748 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2024-01-19T01:11:10.024 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-01-19T01:11:10.257 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-01-19T01:11:11.906 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-01-19T01:11:12.098 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-01-19T01:11:14.882 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-01-19T01:11:15.257 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 19 MB 00:02 2024-01-19T01:11:19.008 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 5.9 MB 00:00 2024-01-19T01:11:19.166 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 13 MB/s | 5.9 MB 00:00 2024-01-19T01:11:20.400 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2024-01-19T01:11:21.700 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2024-01-19T01:11:22.446 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2024-01-19T01:11:23.876 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 20 MB 00:02 2024-01-19T01:11:27.282 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-01-19T01:11:28.667 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-01-19T01:11:29.839 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 14 M 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.1 M 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:17.2.7-268.g6441305d.el9 ceph 5.0 M 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 23 M 2024-01-19T01:11:29.842 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 634 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 458 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 4.9 M 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:17.2.7-268.g6441305d.el9 ceph 45 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 116 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:17.2.7-268.g6441305d.el9 ceph 156 k 2024-01-19T01:11:29.843 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:17.2.7-268.g6441305d.el9 ceph 319 k 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 293 k 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 99 k 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout:Install 22 Packages 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-01-19T01:11:29.844 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:29.845 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 58 M 2024-01-19T01:11:29.845 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:11:30.672 INFO:teuthology.orchestra.run.smithi152.stdout:(1/24): ceph-base-17.2.7-268.g6441305d.el9.x86_ 7.7 MB/s | 5.0 MB 00:00 2024-01-19T01:11:30.756 INFO:teuthology.orchestra.run.smithi152.stdout:(2/24): ceph-selinux-17.2.7-268.g6441305d.el9.x 301 kB/s | 25 kB 00:00 2024-01-19T01:11:30.864 INFO:teuthology.orchestra.run.smithi152.stdout:(3/24): libcephfs2-17.2.7-268.g6441305d.el9.x86 5.7 MB/s | 634 kB 00:00 2024-01-19T01:11:31.014 INFO:teuthology.orchestra.run.smithi152.stdout:(4/24): libradosstriper1-17.2.7-268.g6441305d.e 3.0 MB/s | 458 kB 00:00 2024-01-19T01:11:31.208 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 14 M 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrading: 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout: librados2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.1 M 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T01:11:31.219 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base x86_64 2:17.2.7-268.g6441305d.el9 ceph 5.0 M 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 23 M 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 634 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1 x86_64 2:17.2.7-268.g6441305d.el9 ceph 458 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2 x86_64 2:17.2.7-268.g6441305d.el9 ceph 4.9 M 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-19T01:11:31.220 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse x86_64 2:17.2.7-268.g6441305d.el9 ceph 45 k 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common x86_64 2:17.2.7-268.g6441305d.el9 ceph 116 k 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs x86_64 2:17.2.7-268.g6441305d.el9 ceph 156 k 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados x86_64 2:17.2.7-268.g6441305d.el9 ceph 319 k 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 293 k 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw x86_64 2:17.2.7-268.g6441305d.el9 ceph 99 k 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout:Install 22 Packages 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout:Upgrade 2 Packages 2024-01-19T01:11:31.221 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:31.222 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 58 M 2024-01-19T01:11:31.222 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:11:31.265 INFO:teuthology.orchestra.run.smithi152.stdout:(5/24): ceph-radosgw-17.2.7-268.g6441305d.el9.x 11 MB/s | 14 MB 00:01 2024-01-19T01:11:31.373 INFO:teuthology.orchestra.run.smithi152.stdout:(6/24): python3-ceph-argparse-17.2.7-268.g64413 417 kB/s | 45 kB 00:00 2024-01-19T01:11:31.449 INFO:teuthology.orchestra.run.smithi152.stdout:(7/24): librgw2-17.2.7-268.g6441305d.el9.x86_64 11 MB/s | 4.9 MB 00:00 2024-01-19T01:11:31.474 INFO:teuthology.orchestra.run.smithi152.stdout:(8/24): python3-ceph-common-17.2.7-268.g6441305 1.1 MB/s | 116 kB 00:00 2024-01-19T01:11:31.499 INFO:teuthology.orchestra.run.smithi152.stdout:(9/24): python3-cephfs-17.2.7-268.g6441305d.el9 3.0 MB/s | 156 kB 00:00 2024-01-19T01:11:31.524 INFO:teuthology.orchestra.run.smithi152.stdout:(10/24): python3-rados-17.2.7-268.g6441305d.el9 6.2 MB/s | 319 kB 00:00 2024-01-19T01:11:31.558 INFO:teuthology.orchestra.run.smithi152.stdout:(11/24): python3-rbd-17.2.7-268.g6441305d.el9.x 4.9 MB/s | 293 kB 00:00 2024-01-19T01:11:31.583 INFO:teuthology.orchestra.run.smithi152.stdout:(12/24): python3-rgw-17.2.7-268.g6441305d.el9.x 1.7 MB/s | 99 kB 00:00 2024-01-19T01:11:31.692 INFO:teuthology.orchestra.run.smithi152.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 408 kB/s | 44 kB 00:00 2024-01-19T01:11:31.717 INFO:teuthology.orchestra.run.smithi152.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-01-19T01:11:31.793 INFO:teuthology.orchestra.run.smithi152.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-01-19T01:11:31.818 INFO:teuthology.orchestra.run.smithi152.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-01-19T01:11:31.835 INFO:teuthology.orchestra.run.smithi152.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 995 kB/s | 41 kB 00:00 2024-01-19T01:11:32.250 INFO:teuthology.orchestra.run.smithi152.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 117 kB/s | 49 kB 00:00 2024-01-19T01:11:32.544 INFO:teuthology.orchestra.run.smithi152.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 229 kB/s | 67 kB 00:00 2024-01-19T01:11:33.027 INFO:teuthology.orchestra.run.smithi152.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 255 kB/s | 309 kB 00:01 2024-01-19T01:11:33.201 INFO:teuthology.orchestra.run.smithi079.stdout:(1/24): ceph-common-17.2.7-268.g6441305d.el9.x8 12 MB/s | 23 MB 00:01 2024-01-19T01:11:33.268 INFO:teuthology.orchestra.run.smithi079.stdout:(2/24): ceph-base-17.2.7-268.g6441305d.el9.x86_ 2.7 MB/s | 5.0 MB 00:01 2024-01-19T01:11:33.301 INFO:teuthology.orchestra.run.smithi079.stdout:(3/24): ceph-selinux-17.2.7-268.g6441305d.el9.x 252 kB/s | 25 kB 00:00 2024-01-19T01:11:33.361 INFO:teuthology.orchestra.run.smithi152.stdout:(21/24): librados2-17.2.7-268.g6441305d.el9.x86 9.2 MB/s | 3.1 MB 00:00 2024-01-19T01:11:33.377 INFO:teuthology.orchestra.run.smithi079.stdout:(4/24): libradosstriper1-17.2.7-268.g6441305d.e 6.0 MB/s | 458 kB 00:00 2024-01-19T01:11:33.452 INFO:teuthology.orchestra.run.smithi079.stdout:(5/24): libcephfs2-17.2.7-268.g6441305d.el9.x86 3.4 MB/s | 634 kB 00:00 2024-01-19T01:11:33.561 INFO:teuthology.orchestra.run.smithi079.stdout:(6/24): python3-ceph-argparse-17.2.7-268.g64413 415 kB/s | 45 kB 00:00 2024-01-19T01:11:33.670 INFO:teuthology.orchestra.run.smithi079.stdout:(7/24): python3-ceph-common-17.2.7-268.g6441305 1.0 MB/s | 116 kB 00:00 2024-01-19T01:11:33.778 INFO:teuthology.orchestra.run.smithi152.stdout:(22/24): librbd1-17.2.7-268.g6441305d.el9.x86_6 7.0 MB/s | 2.9 MB 00:00 2024-01-19T01:11:33.779 INFO:teuthology.orchestra.run.smithi079.stdout:(8/24): python3-cephfs-17.2.7-268.g6441305d.el9 1.4 MB/s | 156 kB 00:00 2024-01-19T01:11:33.879 INFO:teuthology.orchestra.run.smithi079.stdout:(9/24): python3-rados-17.2.7-268.g6441305d.el9. 3.1 MB/s | 319 kB 00:00 2024-01-19T01:11:33.979 INFO:teuthology.orchestra.run.smithi079.stdout:(10/24): python3-rbd-17.2.7-268.g6441305d.el9.x 2.9 MB/s | 293 kB 00:00 2024-01-19T01:11:34.046 INFO:teuthology.orchestra.run.smithi079.stdout:(11/24): librgw2-17.2.7-268.g6441305d.el9.x86_6 7.3 MB/s | 4.9 MB 00:00 2024-01-19T01:11:34.071 INFO:teuthology.orchestra.run.smithi079.stdout:(12/24): python3-rgw-17.2.7-268.g6441305d.el9.x 1.1 MB/s | 99 kB 00:00 2024-01-19T01:11:34.163 INFO:teuthology.orchestra.run.smithi079.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 481 kB/s | 44 kB 00:00 2024-01-19T01:11:34.230 INFO:teuthology.orchestra.run.smithi079.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 866 kB/s | 159 kB 00:00 2024-01-19T01:11:34.255 INFO:teuthology.orchestra.run.smithi079.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-01-19T01:11:34.280 INFO:teuthology.orchestra.run.smithi079.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-01-19T01:11:34.297 INFO:teuthology.orchestra.run.smithi079.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2024-01-19T01:11:34.362 INFO:teuthology.orchestra.run.smithi152.stdout:(23/24): ceph-common-17.2.7-268.g6441305d.el9.x 5.2 MB/s | 23 MB 00:04 2024-01-19T01:11:34.515 INFO:teuthology.orchestra.run.smithi079.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 224 kB/s | 49 kB 00:00 2024-01-19T01:11:34.648 INFO:teuthology.orchestra.run.smithi079.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 506 kB/s | 67 kB 00:00 2024-01-19T01:11:34.673 INFO:teuthology.orchestra.run.smithi079.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 787 kB/s | 309 kB 00:00 2024-01-19T01:11:34.932 INFO:teuthology.orchestra.run.smithi079.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-01-19T01:11:35.216 INFO:teuthology.orchestra.run.smithi079.stdout:(22/24): ceph-radosgw-17.2.7-268.g6441305d.el9. 3.7 MB/s | 14 MB 00:03 2024-01-19T01:11:35.274 INFO:teuthology.orchestra.run.smithi079.stdout:(23/24): librados2-17.2.7-268.g6441305d.el9.x86 5.1 MB/s | 3.1 MB 00:00 2024-01-19T01:11:35.333 INFO:teuthology.orchestra.run.smithi079.stdout:(24/24): librbd1-17.2.7-268.g6441305d.el9.x86_6 7.3 MB/s | 2.9 MB 00:00 2024-01-19T01:11:35.336 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:11:35.336 INFO:teuthology.orchestra.run.smithi079.stdout:Total 14 MB/s | 58 MB 00:04 2024-01-19T01:11:35.337 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:11:35.366 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:11:35.366 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:11:35.835 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:11:35.835 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:11:36.179 INFO:teuthology.orchestra.run.smithi152.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 446 kB/s | 1.6 MB 00:03 2024-01-19T01:11:36.183 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:11:36.183 INFO:teuthology.orchestra.run.smithi152.stdout:Total 9.1 MB/s | 58 MB 00:06 2024-01-19T01:11:36.184 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:11:36.216 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:11:36.216 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:11:36.694 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:11:36.694 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:11:37.088 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:11:37.421 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-01-19T01:11:37.485 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T01:11:37.568 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T01:11:37.654 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T01:11:37.687 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T01:11:37.774 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T01:11:37.820 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-01-19T01:11:37.858 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-01-19T01:11:37.904 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:11:38.249 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-01-19T01:11:38.289 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-01-19T01:11:38.322 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T01:11:38.331 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T01:11:38.403 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T01:11:38.419 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T01:11:38.438 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-01-19T01:11:38.481 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 10/26 2024-01-19T01:11:38.509 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T01:11:38.520 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T01:11:38.540 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T01:11:38.589 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T01:11:38.632 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T01:11:38.633 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T01:11:38.675 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-01-19T01:11:38.682 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T01:11:38.703 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-01-19T01:11:38.737 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-01-19T01:11:38.800 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-01-19T01:11:38.833 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-01-19T01:11:39.126 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-01-19T01:11:39.137 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-01-19T01:11:39.163 INFO:teuthology.orchestra.run.smithi079.stdout: Upgrading : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T01:11:39.200 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T01:11:39.273 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T01:11:39.276 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 8/26 2024-01-19T01:11:39.310 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-01-19T01:11:39.342 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 19/26 2024-01-19T01:11:39.345 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 10/26 2024-01-19T01:11:39.373 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 20/26 2024-01-19T01:11:39.387 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T01:11:39.453 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T01:11:39.480 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T01:11:39.529 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T01:11:39.569 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-01-19T01:11:39.630 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-01-19T01:11:39.662 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-01-19T01:11:39.954 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-01-19T01:11:39.984 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T01:11:40.081 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 18/26 2024-01-19T01:11:40.155 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 19/26 2024-01-19T01:11:40.187 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 20/26 2024-01-19T01:11:41.755 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T01:11:41.786 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T01:11:42.326 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T01:11:42.364 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T01:11:42.454 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T01:11:42.455 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-19T01:11:42.455 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-19T01:11:42.455 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:42.490 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T01:11:42.579 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T01:11:42.634 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T01:11:43.184 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 21/26 2024-01-19T01:11:43.221 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T01:11:43.330 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 22/26 2024-01-19T01:11:43.330 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-19T01:11:43.330 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-19T01:11:43.331 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:43.363 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T01:11:51.827 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /sys 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /proc 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /mnt 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /var/tmp 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /home 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /root 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout:skipping the directory /tmp 2024-01-19T01:11:51.828 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-01-19T01:11:52.576 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:53.124 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T01:11:53.202 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T01:11:53.202 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:11:53.202 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-19T01:11:53.203 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T01:11:53.203 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T01:11:53.203 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:53.204 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T01:11:53.286 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T01:11:53.286 INFO:teuthology.orchestra.run.smithi079.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T01:11:53.858 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T01:11:53.927 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 24/26 2024-01-19T01:11:54.044 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:11:54.044 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-19T01:11:54.044 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T01:11:54.044 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T01:11:54.044 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:54.045 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T01:11:54.045 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-01-19T01:11:54.045 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 1/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 5/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 6/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 7/26 2024-01-19T01:11:54.561 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 8/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 9/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 10/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-01-19T01:11:54.562 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T01:11:54.563 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-01-19T01:11:54.563 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 25/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 1/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 3/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 4/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 5/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 6/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:17.2.7-268.g6441305d.el9.x86_64 7/26 2024-01-19T01:11:55.199 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x 8/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86 9/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 10/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 11/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 12/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 13/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-01-19T01:11:55.200 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-01-19T01:11:55.201 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:17.2.7-268.g6441305d.el9.x86_64 23/26 2024-01-19T01:11:55.201 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-01-19T01:11:55.201 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:17.2.7-268.g6441305d.el9.x86_64 25/26 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout:Upgraded: 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.600 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-19T01:11:55.601 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.602 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.602 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:55.602 INFO:teuthology.orchestra.run.smithi079.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-19T01:11:55.602 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:55.602 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:11:55.964 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-test 2024-01-19T01:11:56.108 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-01-19T01:11:56.108 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:56.108 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-01-19T01:11:56.108 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.108 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.108 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:11:56.108 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-19T01:11:56.109 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.110 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-19T01:11:56.111 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.111 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.111 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:11:56.111 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-19T01:11:56.111 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:56.111 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:11:56.520 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:28 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:11:56.530 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-01-19T01:11:56.673 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test x86_64 2:17.2.7-268.g6441305d.el9 ceph 48 M 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite x86_64 2:17.2.7-268.g6441305d.el9 ceph 162 k 2024-01-19T01:11:56.674 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:11:56.675 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 49 M 2024-01-19T01:11:56.676 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 195 M 2024-01-19T01:11:56.676 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:11:56.889 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-01-19T01:11:56.956 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): libcephsqlite-17.2.7-268.g6441305d.el9.x 581 kB/s | 162 kB 00:00 2024-01-19T01:11:56.981 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 682 kB/s | 63 kB 00:00 2024-01-19T01:11:57.023 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 186 kB 00:00 2024-01-19T01:11:57.048 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-01-19T01:11:57.112 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:30 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:11:57.270 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:11:57.271 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:11:57.271 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:11:57.271 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:17.2.7-268.g6441305d.el9 ceph 48 M 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:17.2.7-268.g6441305d.el9 ceph 162 k 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-01-19T01:11:57.272 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:11:57.273 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 49 M 2024-01-19T01:11:57.273 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 195 M 2024-01-19T01:11:57.273 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:11:57.698 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 707 kB/s | 300 kB 00:00 2024-01-19T01:11:57.765 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 940 kB/s | 63 kB 00:00 2024-01-19T01:11:57.807 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): libcephsqlite-17.2.7-268.g6441305d.el9.x 305 kB/s | 162 kB 00:00 2024-01-19T01:11:57.899 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 186 kB 00:00 2024-01-19T01:11:57.999 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-01-19T01:12:07.947 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): ceph-test-17.2.7-268.g6441305d.el9.x86_6 4.3 MB/s | 48 MB 00:11 2024-01-19T01:12:07.949 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:07.950 INFO:teuthology.orchestra.run.smithi079.stdout:Total 4.3 MB/s | 49 MB 00:11 2024-01-19T01:12:07.950 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:07.966 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:07.966 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:08.352 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:08.352 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:08.949 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:09.012 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-19T01:12:09.046 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-19T01:12:09.098 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-19T01:12:09.147 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-19T01:12:09.175 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T01:12:09.740 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): ceph-test-17.2.7-268.g6441305d.el9.x86_6 3.8 MB/s | 48 MB 00:12 2024-01-19T01:12:09.742 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:09.742 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.9 MB/s | 49 MB 00:12 2024-01-19T01:12:09.743 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:12:09.759 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:12:09.760 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:12:10.142 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:12:10.142 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:12:10.759 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:12:10.815 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-19T01:12:10.848 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-19T01:12:10.892 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-19T01:12:10.932 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-19T01:12:10.960 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T01:12:13.646 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T01:12:13.723 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T01:12:15.455 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 5/6 2024-01-19T01:12:15.618 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T01:12:15.984 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T01:12:15.984 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 1/6 2024-01-19T01:12:15.985 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2/6 2024-01-19T01:12:15.985 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-19T01:12:15.985 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-19T01:12:15.985 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-19T01:12:16.372 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: jq-1.6-16.el9.x86_64 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:16.373 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:16.674 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph 2024-01-19T01:12:17.229 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:00:49 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:17.384 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:17.386 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-01-19T01:12:17.386 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:12:17.386 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-01-19T01:12:17.386 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: ceph x86_64 2:17.2.7-268.g6441305d.el9 ceph 6.5 k 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.1 M 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr x86_64 2:17.2.7-268.g6441305d.el9 ceph 1.4 M 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 239 k 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.7 M 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd x86_64 2:17.2.7-268.g6441305d.el9 ceph 15 M 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-19T01:12:17.387 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-19T01:12:17.388 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================== 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout:Install 33 Packages 2024-01-19T01:12:17.389 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:17.390 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 27 M 2024-01-19T01:12:17.390 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 99 M 2024-01-19T01:12:17.390 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:12:17.699 INFO:teuthology.orchestra.run.smithi079.stdout:(1/33): ceph-17.2.7-268.g6441305d.el9.x86_64.rp 43 kB/s | 6.5 kB 00:00 2024-01-19T01:12:17.915 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 6/6 2024-01-19T01:12:17.915 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 1/6 2024-01-19T01:12:17.915 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2/6 2024-01-19T01:12:17.915 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-19T01:12:17.915 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-19T01:12:17.915 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-19T01:12:17.950 INFO:teuthology.orchestra.run.smithi079.stdout:(2/33): ceph-mgr-17.2.7-268.g6441305d.el9.x86_6 3.5 MB/s | 1.4 MB 00:00 2024-01-19T01:12:18.200 INFO:teuthology.orchestra.run.smithi079.stdout:(3/33): ceph-mon-17.2.7-268.g6441305d.el9.x86_6 7.4 MB/s | 3.7 MB 00:00 2024-01-19T01:12:18.250 INFO:teuthology.orchestra.run.smithi079.stdout:(4/33): ceph-mds-17.2.7-268.g6441305d.el9.x86_6 2.9 MB/s | 2.1 MB 00:00 2024-01-19T01:12:18.275 INFO:teuthology.orchestra.run.smithi079.stdout:(5/33): ceph-mgr-modules-core-17.2.7-268.g64413 3.1 MB/s | 239 kB 00:00 2024-01-19T01:12:18.298 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-19T01:12:18.298 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:18.298 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:12:18.298 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:18.298 INFO:teuthology.orchestra.run.smithi152.stdout: jq-1.6-16.el9.x86_64 2024-01-19T01:12:18.298 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:18.299 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-19T01:12:18.299 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-19T01:12:18.299 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-19T01:12:18.299 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:18.299 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:18.384 INFO:teuthology.orchestra.run.smithi079.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 371 kB/s | 32 kB 00:00 2024-01-19T01:12:18.409 INFO:teuthology.orchestra.run.smithi079.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 908 kB/s | 144 kB 00:00 2024-01-19T01:12:18.434 INFO:teuthology.orchestra.run.smithi079.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-01-19T01:12:18.460 INFO:teuthology.orchestra.run.smithi079.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-01-19T01:12:18.501 INFO:teuthology.orchestra.run.smithi079.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-01-19T01:12:18.560 INFO:teuthology.orchestra.run.smithi079.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-01-19T01:12:18.585 INFO:teuthology.orchestra.run.smithi079.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-01-19T01:12:18.611 INFO:teuthology.orchestra.run.smithi079.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-19T01:12:18.612 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-01-19T01:12:18.636 INFO:teuthology.orchestra.run.smithi079.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-19T01:12:18.662 INFO:teuthology.orchestra.run.smithi079.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-19T01:12:18.929 INFO:teuthology.orchestra.run.smithi079.stdout:(16/33): ceph-osd-17.2.7-268.g6441305d.el9.x86_ 16 MB/s | 15 MB 00:00 2024-01-19T01:12:19.129 INFO:teuthology.orchestra.run.smithi079.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 88 kB/s | 43 kB 00:00 2024-01-19T01:12:19.171 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:52 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:19.238 INFO:teuthology.orchestra.run.smithi079.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-01-19T01:12:19.263 INFO:teuthology.orchestra.run.smithi079.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 286 kB/s | 172 kB 00:00 2024-01-19T01:12:19.329 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:17.2.7-268.g6441305d.el9 ceph 6.5 k 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.1 M 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:17.2.7-268.g6441305d.el9 ceph 1.4 M 2024-01-19T01:12:19.332 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 239 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:17.2.7-268.g6441305d.el9 ceph 3.7 M 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:17.2.7-268.g6441305d.el9 ceph 15 M 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-19T01:12:19.333 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-19T01:12:19.334 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-19T01:12:19.335 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-19T01:12:19.335 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-19T01:12:19.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:19.335 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:12:19.335 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================== 2024-01-19T01:12:19.335 INFO:teuthology.orchestra.run.smithi152.stdout:Install 33 Packages 2024-01-19T01:12:19.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:19.336 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 27 M 2024-01-19T01:12:19.336 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 99 M 2024-01-19T01:12:19.336 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:12:19.346 INFO:teuthology.orchestra.run.smithi079.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-01-19T01:12:19.372 INFO:teuthology.orchestra.run.smithi079.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-01-19T01:12:19.472 INFO:teuthology.orchestra.run.smithi079.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-01-19T01:12:19.497 INFO:teuthology.orchestra.run.smithi079.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2024-01-19T01:12:19.531 INFO:teuthology.orchestra.run.smithi079.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 598 kB/s | 358 kB 00:00 2024-01-19T01:12:19.614 INFO:teuthology.orchestra.run.smithi079.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 674 kB/s | 79 kB 00:00 2024-01-19T01:12:19.639 INFO:teuthology.orchestra.run.smithi079.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 277 kB/s | 46 kB 00:00 2024-01-19T01:12:19.673 INFO:teuthology.orchestra.run.smithi079.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-01-19T01:12:19.740 INFO:teuthology.orchestra.run.smithi079.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2024-01-19T01:12:19.765 INFO:teuthology.orchestra.run.smithi079.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2024-01-19T01:12:19.791 INFO:teuthology.orchestra.run.smithi079.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 305 kB/s | 36 kB 00:00 2024-01-19T01:12:19.883 INFO:teuthology.orchestra.run.smithi079.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-01-19T01:12:19.991 INFO:teuthology.orchestra.run.smithi079.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-01-19T01:12:20.016 INFO:teuthology.orchestra.run.smithi079.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 832 kB/s | 230 kB 00:00 2024-01-19T01:12:20.023 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:20.024 INFO:teuthology.orchestra.run.smithi079.stdout:Total 10 MB/s | 27 MB 00:02 2024-01-19T01:12:20.024 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:20.061 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:20.061 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:20.411 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:20.412 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:21.613 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:21.686 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-01-19T01:12:21.718 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-01-19T01:12:21.760 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-01-19T01:12:21.850 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-01-19T01:12:21.920 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-01-19T01:12:21.962 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-01-19T01:12:22.033 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-01-19T01:12:22.096 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-01-19T01:12:22.135 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2024-01-19T01:12:22.183 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-01-19T01:12:22.238 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-01-19T01:12:22.431 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-01-19T01:12:22.496 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-01-19T01:12:22.534 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-01-19T01:12:22.568 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-01-19T01:12:22.615 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-01-19T01:12:22.691 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-01-19T01:12:22.725 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T01:12:22.759 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-01-19T01:12:22.799 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-01-19T01:12:22.834 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-01-19T01:12:22.877 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-01-19T01:12:22.965 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-01-19T01:12:22.999 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-01-19T01:12:23.071 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-01-19T01:12:23.217 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-01-19T01:12:23.340 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T01:12:23.489 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 28/33 2024-01-19T01:12:23.513 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T01:12:23.538 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T01:12:23.538 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:23.538 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-19T01:12:23.538 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T01:12:23.538 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T01:12:23.538 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:24.876 INFO:teuthology.orchestra.run.smithi152.stdout:(1/33): ceph-17.2.7-268.g6441305d.el9.x86_64.rp 31 kB/s | 6.5 kB 00:00 2024-01-19T01:12:24.882 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T01:12:24.909 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T01:12:24.909 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:24.909 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-19T01:12:24.909 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T01:12:24.909 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T01:12:24.909 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi152.stdout:(2/33): ceph-mgr-17.2.7-268.g6441305d.el9.x86_6 4.2 MB/s | 1.4 MB 00:00 2024-01-19T01:12:25.283 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T01:12:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T01:12:25.306 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:25.306 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-19T01:12:25.306 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T01:12:25.306 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T01:12:25.306 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:25.369 INFO:teuthology.orchestra.run.smithi152.stdout:(3/33): ceph-mds-17.2.7-268.g6441305d.el9.x86_6 2.9 MB/s | 2.1 MB 00:00 2024-01-19T01:12:25.477 INFO:teuthology.orchestra.run.smithi152.stdout:(4/33): ceph-mgr-modules-core-17.2.7-268.g64413 2.2 MB/s | 239 kB 00:00 2024-01-19T01:12:25.500 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T01:12:25.523 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T01:12:25.523 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:25.523 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-19T01:12:25.524 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T01:12:25.524 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T01:12:25.524 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:25.581 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T01:12:25.586 INFO:teuthology.orchestra.run.smithi152.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-01-19T01:12:25.611 INFO:teuthology.orchestra.run.smithi152.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-01-19T01:12:25.636 INFO:teuthology.orchestra.run.smithi152.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-01-19T01:12:25.670 INFO:teuthology.orchestra.run.smithi152.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-01-19T01:12:25.720 INFO:teuthology.orchestra.run.smithi152.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-01-19T01:12:25.779 INFO:teuthology.orchestra.run.smithi152.stdout:(10/33): ceph-mon-17.2.7-268.g6441305d.el9.x86_ 4.1 MB/s | 3.7 MB 00:00 2024-01-19T01:12:25.812 INFO:teuthology.orchestra.run.smithi152.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-01-19T01:12:25.838 INFO:teuthology.orchestra.run.smithi152.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-01-19T01:12:25.863 INFO:teuthology.orchestra.run.smithi152.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-19T01:12:25.888 INFO:teuthology.orchestra.run.smithi152.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-19T01:12:25.914 INFO:teuthology.orchestra.run.smithi152.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-19T01:12:26.073 INFO:teuthology.orchestra.run.smithi152.stdout:(16/33): ceph-osd-17.2.7-268.g6441305d.el9.x86_ 14 MB/s | 15 MB 00:01 2024-01-19T01:12:26.190 INFO:teuthology.orchestra.run.smithi152.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-01-19T01:12:26.240 INFO:teuthology.orchestra.run.smithi152.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-01-19T01:12:26.265 INFO:teuthology.orchestra.run.smithi152.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 489 kB/s | 172 kB 00:00 2024-01-19T01:12:26.291 INFO:teuthology.orchestra.run.smithi152.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-01-19T01:12:26.324 INFO:teuthology.orchestra.run.smithi152.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-01-19T01:12:26.349 INFO:teuthology.orchestra.run.smithi152.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-01-19T01:12:26.375 INFO:teuthology.orchestra.run.smithi152.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-01-19T01:12:26.400 INFO:teuthology.orchestra.run.smithi152.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-01-19T01:12:26.425 INFO:teuthology.orchestra.run.smithi152.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-01-19T01:12:26.442 INFO:teuthology.orchestra.run.smithi152.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-01-19T01:12:26.484 INFO:teuthology.orchestra.run.smithi152.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-01-19T01:12:26.509 INFO:teuthology.orchestra.run.smithi152.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-01-19T01:12:26.535 INFO:teuthology.orchestra.run.smithi152.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 973 kB/s | 90 kB 00:00 2024-01-19T01:12:26.560 INFO:teuthology.orchestra.run.smithi152.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-01-19T01:12:26.585 INFO:teuthology.orchestra.run.smithi152.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-01-19T01:12:26.619 INFO:teuthology.orchestra.run.smithi152.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-01-19T01:12:26.652 INFO:teuthology.orchestra.run.smithi152.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-01-19T01:12:26.656 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:26.656 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.6 MB/s | 27 MB 00:07 2024-01-19T01:12:26.656 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:12:26.695 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:12:26.695 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:12:27.037 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:12:27.037 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:12:27.282 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T01:12:27.282 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-2:17.2.7-268.g6441305d.el9.x86_64 1/33 2024-01-19T01:12:27.282 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2/33 2024-01-19T01:12:27.282 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 3/33 2024-01-19T01:12:27.282 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 4/33 2024-01-19T01:12:27.282 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 5/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 6/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-01-19T01:12:27.283 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-01-19T01:12:27.284 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-01-19T01:12:27.285 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-01-19T01:12:27.285 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-01-19T01:12:28.259 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-01-19T01:12:28.259 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-19T01:12:28.260 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-ply-3.11-14.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-19T01:12:28.261 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:28.262 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:28.276 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:12:28.331 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-01-19T01:12:28.363 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-01-19T01:12:28.396 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-01-19T01:12:28.488 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-01-19T01:12:28.566 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-01-19T01:12:28.574 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-base 2024-01-19T01:12:28.607 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-01-19T01:12:28.679 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-01-19T01:12:28.741 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-01-19T01:12:28.779 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2024-01-19T01:12:28.828 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-01-19T01:12:28.884 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-01-19T01:12:29.076 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-01-19T01:12:29.125 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-01-19T01:12:29.136 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:01 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:29.164 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-01-19T01:12:29.199 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-01-19T01:12:29.229 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-01-19T01:12:29.237 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:12:29.287 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:29.288 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:12:29.288 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:29.305 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-01-19T01:12:29.339 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T01:12:29.354 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install cephadm 2024-01-19T01:12:29.378 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-01-19T01:12:29.419 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-01-19T01:12:29.462 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-01-19T01:12:29.497 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-01-19T01:12:29.586 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-01-19T01:12:29.620 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-01-19T01:12:29.691 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-01-19T01:12:29.829 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-01-19T01:12:29.902 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:01 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:29.957 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T01:12:30.057 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 80 k 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:12:30.058 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:30.059 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-01-19T01:12:30.059 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:30.059 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 80 k 2024-01-19T01:12:30.059 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 364 k 2024-01-19T01:12:30.059 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:12:30.112 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 28/33 2024-01-19T01:12:30.142 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T01:12:30.168 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 29/33 2024-01-19T01:12:30.168 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:30.168 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-19T01:12:30.168 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T01:12:30.168 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T01:12:30.168 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:30.376 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm-17.2.7-268.g6441305d.el9.noarch.rpm 248 kB/s | 80 kB 00:00 2024-01-19T01:12:30.378 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:30.378 INFO:teuthology.orchestra.run.smithi079.stdout:Total 247 kB/s | 80 kB 00:00 2024-01-19T01:12:30.378 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:30.382 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:30.382 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:30.384 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:30.387 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:30.454 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:30.900 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:30.997 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:31.451 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:31.518 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T01:12:31.544 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 30/33 2024-01-19T01:12:31.544 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:31.544 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-19T01:12:31.544 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T01:12:31.544 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T01:12:31.545 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:31.744 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:31.744 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:31.744 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:12:31.744 INFO:teuthology.orchestra.run.smithi079.stdout: cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:31.745 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:31.745 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:31.919 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T01:12:31.942 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 31/33 2024-01-19T01:12:31.942 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:31.942 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-19T01:12:31.942 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T01:12:31.942 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T01:12:31.942 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:31.974 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-immutable-object-cache 2024-01-19T01:12:32.145 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T01:12:32.168 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 32/33 2024-01-19T01:12:32.168 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:32.168 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-19T01:12:32.168 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T01:12:32.168 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T01:12:32.168 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:32.232 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T01:12:32.535 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:04 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:32.689 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:32.690 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:32.690 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-01-19T01:12:32.690 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-268.g6441305d.el9 ceph 137 k 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 137 k 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 379 k 2024-01-19T01:12:32.691 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:12:32.918 INFO:teuthology.orchestra.run.smithi079.stdout:ceph-immutable-object-cache-17.2.7-268.g6441305 604 kB/s | 137 kB 00:00 2024-01-19T01:12:32.919 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:32.919 INFO:teuthology.orchestra.run.smithi079.stdout:Total 600 kB/s | 137 kB 00:00 2024-01-19T01:12:32.919 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:32.926 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:32.926 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:32.977 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:32.977 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:33.110 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:33.169 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T01:12:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T01:12:33.194 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:33.194 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-19T01:12:33.194 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:33.866 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:17.2.7-268.g6441305d.el9.x86_64 33/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:17.2.7-268.g6441305d.el9.x86_64 1/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 3/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 4/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 5/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.n 6/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-01-19T01:12:33.867 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-01-19T01:12:33.868 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-01-19T01:12:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-01-19T01:12:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-01-19T01:12:33.869 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-01-19T01:12:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-01-19T01:12:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-01-19T01:12:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-01-19T01:12:33.870 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-01-19T01:12:34.144 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T01:12:34.144 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:34.144 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:12:34.144 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:34.145 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:34.145 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:34.407 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr 2024-01-19T01:12:34.796 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-01-19T01:12:34.796 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:34.796 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:12:34.796 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-19T01:12:34.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-19T01:12:34.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-19T01:12:34.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-19T01:12:34.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-19T01:12:34.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-19T01:12:34.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-19T01:12:34.799 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:34.799 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:34.972 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:06 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:35.072 INFO:teuthology.orchestra.run.smithi079.stdout:Package ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:12:35.123 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:35.124 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:12:35.124 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:35.153 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-01-19T01:12:35.194 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-dashboard 2024-01-19T01:12:35.727 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:08 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:35.777 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:07 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:35.827 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:12:35.877 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:35.878 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:12:35.878 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:35.931 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:35.931 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:35.931 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 1.7 M 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 23 k 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 15 k 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout:Installing weak dependencies: 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-19T01:12:35.932 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:35.933 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:12:35.933 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:12:35.933 INFO:teuthology.orchestra.run.smithi079.stdout:Install 7 Packages 2024-01-19T01:12:35.933 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:35.933 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.0 M 2024-01-19T01:12:35.933 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 8.1 M 2024-01-19T01:12:35.933 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:12:35.946 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-01-19T01:12:36.466 INFO:teuthology.orchestra.run.smithi079.stdout:(1/7): ceph-prometheus-alerts-17.2.7-268.g64413 47 kB/s | 15 kB 00:00 2024-01-19T01:12:36.492 INFO:teuthology.orchestra.run.smithi079.stdout:(2/7): ceph-grafana-dashboards-17.2.7-268.g6441 70 kB/s | 23 kB 00:00 2024-01-19T01:12:36.499 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:09 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:36.650 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:36.651 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:36.651 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:12:36.651 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 80 k 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 80 k 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 364 k 2024-01-19T01:12:36.652 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:12:36.742 INFO:teuthology.orchestra.run.smithi079.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 33 kB/s | 9.0 kB 00:00 2024-01-19T01:12:36.901 INFO:teuthology.orchestra.run.smithi079.stdout:(4/7): ceph-mgr-dashboard-17.2.7-268.g6441305d. 2.3 MB/s | 1.7 MB 00:00 2024-01-19T01:12:36.951 INFO:teuthology.orchestra.run.smithi079.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 89 kB/s | 41 kB 00:00 2024-01-19T01:12:36.976 INFO:teuthology.orchestra.run.smithi079.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 131 kB/s | 31 kB 00:00 2024-01-19T01:12:37.054 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-17.2.7-268.g6441305d.el9.noarch.rpm 200 kB/s | 80 kB 00:00 2024-01-19T01:12:37.055 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:37.055 INFO:teuthology.orchestra.run.smithi152.stdout:Total 200 kB/s | 80 kB 00:00 2024-01-19T01:12:37.055 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:12:37.060 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:12:37.060 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:12:37.062 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:12:37.062 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:12:37.139 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:12:37.585 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:37.691 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:37.826 INFO:teuthology.orchestra.run.smithi079.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 204 kB/s | 188 kB 00:00 2024-01-19T01:12:37.829 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:37.829 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-01-19T01:12:37.829 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:37.839 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:37.839 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:37.914 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:37.915 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:38.105 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:38.222 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:38.278 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-01-19T01:12:38.312 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-01-19T01:12:38.367 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-01-19T01:12:38.399 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-01-19T01:12:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/1 2024-01-19T01:12:38.406 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:38.407 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:12:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:38.407 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:38.407 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:38.427 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 5/7 2024-01-19T01:12:38.606 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-01-19T01:12:38.656 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 6/7 2024-01-19T01:12:38.689 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T01:12:39.178 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:12 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:39.257 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T01:12:39.257 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 1/7 2024-01-19T01:12:39.257 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2/7 2024-01-19T01:12:39.257 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 3/7 2024-01-19T01:12:39.257 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-01-19T01:12:39.257 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-01-19T01:12:39.258 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-01-19T01:12:39.334 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-268.g6441305d.el9 ceph 137 k 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:39.335 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-01-19T01:12:39.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:39.336 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 137 k 2024-01-19T01:12:39.336 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 379 k 2024-01-19T01:12:39.336 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:12:39.577 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-01-19T01:12:39.577 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:39.577 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:12:39.577 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:39.577 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:39.577 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:39.578 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-19T01:12:39.578 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-19T01:12:39.578 INFO:teuthology.orchestra.run.smithi079.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-19T01:12:39.578 INFO:teuthology.orchestra.run.smithi079.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-19T01:12:39.578 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:39.578 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:39.684 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-17.2.7-268.g6441305 393 kB/s | 137 kB 00:00 2024-01-19T01:12:39.685 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:39.685 INFO:teuthology.orchestra.run.smithi152.stdout:Total 391 kB/s | 137 kB 00:00 2024-01-19T01:12:39.685 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:12:39.693 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:12:39.693 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:12:39.744 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:12:39.744 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:12:39.813 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-19T01:12:39.880 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:12:39.935 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T01:12:39.959 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T01:12:39.959 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:12:39.959 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-19T01:12:39.959 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:40.361 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:12 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:40.511 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout:=========================================================================================== 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout:=========================================================================================== 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 7.4 M 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-19T01:12:40.512 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout:=========================================================================================== 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout:Install 12 Packages 2024-01-19T01:12:40.513 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:40.514 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 37 M 2024-01-19T01:12:40.514 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 209 M 2024-01-19T01:12:40.514 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:12:40.726 INFO:teuthology.orchestra.run.smithi079.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-01-19T01:12:40.760 INFO:teuthology.orchestra.run.smithi079.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2024-01-19T01:12:40.802 INFO:teuthology.orchestra.run.smithi079.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 34 kB 00:00 2024-01-19T01:12:40.852 INFO:teuthology.orchestra.run.smithi079.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-01-19T01:12:40.937 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-268.g6441305d.e 1/1 2024-01-19T01:12:40.937 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:40.937 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:12:40.937 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:12:40.938 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:40.938 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:41.044 INFO:teuthology.orchestra.run.smithi079.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-01-19T01:12:41.111 INFO:teuthology.orchestra.run.smithi079.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-01-19T01:12:41.137 INFO:teuthology.orchestra.run.smithi079.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-01-19T01:12:41.179 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-01-19T01:12:41.445 INFO:teuthology.orchestra.run.smithi079.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-01-19T01:12:41.479 INFO:teuthology.orchestra.run.smithi079.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 802 kB 00:00 2024-01-19T01:12:41.504 INFO:teuthology.orchestra.run.smithi079.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 194 kB 00:00 2024-01-19T01:12:41.748 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:14 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:41.848 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:12:41.899 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:41.900 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:12:41.900 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:41.972 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-01-19T01:12:42.104 INFO:teuthology.orchestra.run.smithi079.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-2 4.9 MB/s | 7.4 MB 00:01 2024-01-19T01:12:42.346 INFO:teuthology.orchestra.run.smithi079.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 283 kB/s | 245 kB 00:00 2024-01-19T01:12:42.349 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:42.349 INFO:teuthology.orchestra.run.smithi079.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-19T01:12:42.365 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:42.390 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:42.391 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:42.532 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:15 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:42.683 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 1.7 M 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 23 k 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 15 k 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-19T01:12:42.684 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout:Install 7 Packages 2024-01-19T01:12:42.685 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:42.686 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.0 M 2024-01-19T01:12:42.686 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 8.1 M 2024-01-19T01:12:42.686 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:12:42.814 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:42.815 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:43.229 INFO:teuthology.orchestra.run.smithi152.stdout:(1/7): ceph-prometheus-alerts-17.2.7-268.g64413 76 kB/s | 15 kB 00:00 2024-01-19T01:12:43.338 INFO:teuthology.orchestra.run.smithi152.stdout:(2/7): ceph-grafana-dashboards-17.2.7-268.g6441 78 kB/s | 23 kB 00:00 2024-01-19T01:12:43.396 INFO:teuthology.orchestra.run.smithi152.stdout:(3/7): ceph-mgr-dashboard-17.2.7-268.g6441305d. 4.7 MB/s | 1.7 MB 00:00 2024-01-19T01:12:43.472 INFO:teuthology.orchestra.run.smithi152.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 37 kB/s | 9.0 kB 00:00 2024-01-19T01:12:43.519 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:43.619 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-19T01:12:43.650 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-19T01:12:43.705 INFO:teuthology.orchestra.run.smithi152.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 111 kB/s | 41 kB 00:00 2024-01-19T01:12:43.722 INFO:teuthology.orchestra.run.smithi152.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 94 kB/s | 31 kB 00:00 2024-01-19T01:12:43.746 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-19T01:12:43.787 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-19T01:12:43.957 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T01:12:43.964 INFO:teuthology.orchestra.run.smithi152.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 383 kB/s | 188 kB 00:00 2024-01-19T01:12:43.966 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:43.966 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-01-19T01:12:43.966 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:12:43.974 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:12:43.974 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:12:43.986 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T01:12:44.048 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:12:44.048 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:12:44.057 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-19T01:12:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:12:44.415 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-01-19T01:12:44.449 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-01-19T01:12:44.495 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-01-19T01:12:44.527 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-01-19T01:12:44.556 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 5/7 2024-01-19T01:12:44.708 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-19T01:12:44.786 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 6/7 2024-01-19T01:12:44.816 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-19T01:12:44.817 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T01:12:45.378 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 7/7 2024-01-19T01:12:45.378 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.n 1/7 2024-01-19T01:12:45.378 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2/7 2024-01-19T01:12:45.378 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.no 3/7 2024-01-19T01:12:45.378 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-01-19T01:12:45.379 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-01-19T01:12:45.379 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-01-19T01:12:45.717 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-01-19T01:12:45.717 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:45.717 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:12:45.717 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:45.717 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:45.717 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:45.717 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-19T01:12:45.718 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-19T01:12:45.718 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-19T01:12:45.718 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-19T01:12:45.718 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:45.718 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:45.945 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-19T01:12:46.058 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-19T01:12:46.527 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:19 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:46.678 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout:=========================================================================================== 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout:=========================================================================================== 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 7.4 M 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-19T01:12:46.679 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:12:46.680 INFO:teuthology.orchestra.run.smithi152.stdout:=========================================================================================== 2024-01-19T01:12:46.681 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-01-19T01:12:46.681 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:46.681 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 37 M 2024-01-19T01:12:46.681 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 209 M 2024-01-19T01:12:46.682 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:12:46.963 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 200 kB/s | 32 kB 00:00 2024-01-19T01:12:46.996 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2024-01-19T01:12:47.063 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-01-19T01:12:47.088 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 379 kB/s | 34 kB 00:00 2024-01-19T01:12:47.093 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-19T01:12:47.122 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T01:12:47.339 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-01-19T01:12:47.405 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-01-19T01:12:47.431 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-01-19T01:12:47.523 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-26 10 MB/s | 7.4 MB 00:00 2024-01-19T01:12:47.664 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 802 kB 00:00 2024-01-19T01:12:47.740 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 904 kB/s | 194 kB 00:00 2024-01-19T01:12:47.940 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-01-19T01:12:48.090 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 580 kB/s | 245 kB 00:00 2024-01-19T01:12:48.092 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:48.093 INFO:teuthology.orchestra.run.smithi152.stdout:Total 26 MB/s | 37 MB 00:01 2024-01-19T01:12:48.113 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:12:48.138 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:12:48.138 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:12:48.565 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:12:48.565 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:12:49.274 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:12:49.346 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-19T01:12:49.377 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-19T01:12:49.471 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-19T01:12:49.514 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-19T01:12:49.693 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T01:12:49.721 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T01:12:49.803 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-19T01:12:49.847 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T01:12:49.847 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 1/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-19T01:12:49.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-19T01:12:50.336 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-19T01:12:50.336 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:50.336 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:12:50.336 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:50.336 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:50.337 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-19T01:12:50.577 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-19T01:12:50.727 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-rook 2024-01-19T01:12:51.290 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:23 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:51.441 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:51.442 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-01-19T01:12:51.442 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:12:51.442 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-01-19T01:12:51.442 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 47 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-19T01:12:51.443 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-19T01:12:51.444 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-19T01:12:51.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:51.444 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:12:51.444 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-01-19T01:12:51.444 INFO:teuthology.orchestra.run.smithi079.stdout:Install 13 Packages 2024-01-19T01:12:51.444 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:51.444 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.0 M 2024-01-19T01:12:51.445 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 26 M 2024-01-19T01:12:51.445 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:12:51.745 INFO:teuthology.orchestra.run.smithi079.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2024-01-19T01:12:51.770 INFO:teuthology.orchestra.run.smithi079.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 97 kB/s | 18 kB 00:00 2024-01-19T01:12:51.795 INFO:teuthology.orchestra.run.smithi079.stdout:(3/13): ceph-mgr-rook-17.2.7-268.g6441305d.el9. 223 kB/s | 47 kB 00:00 2024-01-19T01:12:51.836 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-19T01:12:51.879 INFO:teuthology.orchestra.run.smithi079.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 174 kB 00:00 2024-01-19T01:12:51.904 INFO:teuthology.orchestra.run.smithi079.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 988 kB/s | 132 kB 00:00 2024-01-19T01:12:51.929 INFO:teuthology.orchestra.run.smithi079.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 211 kB 00:00 2024-01-19T01:12:51.955 INFO:teuthology.orchestra.run.smithi079.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-01-19T01:12:52.055 INFO:teuthology.orchestra.run.smithi079.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-01-19T01:12:52.080 INFO:teuthology.orchestra.run.smithi079.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-01-19T01:12:52.156 INFO:teuthology.orchestra.run.smithi079.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-01-19T01:12:52.206 INFO:teuthology.orchestra.run.smithi079.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 804 kB/s | 202 kB 00:00 2024-01-19T01:12:52.248 INFO:teuthology.orchestra.run.smithi079.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 977 kB/s | 90 kB 00:00 2024-01-19T01:12:52.523 INFO:teuthology.orchestra.run.smithi079.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.2 MB/s | 1.0 MB 00:00 2024-01-19T01:12:52.524 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:52.525 INFO:teuthology.orchestra.run.smithi079.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-01-19T01:12:52.525 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:52.537 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:52.537 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:52.633 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:52.634 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:52.865 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-19T01:12:52.895 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T01:12:53.151 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:53.318 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-19T01:12:53.382 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-19T01:12:53.438 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-19T01:12:53.476 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-19T01:12:53.517 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-19T01:12:53.611 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-19T01:12:53.727 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-19T01:12:53.778 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-19T01:12:54.492 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-19T01:12:54.542 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-19T01:12:54.576 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-19T01:12:54.621 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-19T01:12:54.646 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 12/12 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-268.g644130 1/12 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T01:12:55.447 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-19T01:12:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-19T01:12:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-19T01:12:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-19T01:12:55.448 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-19T01:12:55.482 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T01:12:55.482 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 1/13 2024-01-19T01:12:55.482 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-19T01:12:55.482 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-19T01:12:55.482 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-19T01:12:55.483 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-19T01:12:55.847 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-19T01:12:55.848 INFO:teuthology.orchestra.run.smithi079.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-19T01:12:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-19T01:12:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-19T01:12:55.849 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:55.849 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:12:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-19T01:12:55.905 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-19T01:12:55.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-19T01:12:55.907 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:55.907 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:12:56.099 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-mgr-cephadm 2024-01-19T01:12:56.261 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-01-19T01:12:56.657 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:28 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:12:56.809 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 121 k 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-19T01:12:56.810 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-19T01:12:56.811 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:56.811 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:12:56.811 INFO:teuthology.orchestra.run.smithi079.stdout:======================================================================================= 2024-01-19T01:12:56.811 INFO:teuthology.orchestra.run.smithi079.stdout:Install 6 Packages 2024-01-19T01:12:56.811 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:12:56.811 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 6.8 M 2024-01-19T01:12:56.811 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 33 M 2024-01-19T01:12:56.812 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:12:56.836 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:29 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:12:56.988 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:12:56.989 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 47 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-19T01:12:56.990 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-01-19T01:12:56.991 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:12:56.992 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.0 M 2024-01-19T01:12:56.992 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-01-19T01:12:56.992 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:12:57.266 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 162 kB/s | 18 kB 00:00 2024-01-19T01:12:57.291 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 178 kB/s | 24 kB 00:00 2024-01-19T01:12:57.337 INFO:teuthology.orchestra.run.smithi079.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-19T01:12:57.341 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-01-19T01:12:57.366 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-01-19T01:12:57.391 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-19T01:12:57.404 INFO:teuthology.orchestra.run.smithi079.stdout:(2/6): ceph-mgr-cephadm-17.2.7-268.g6441305d.el 495 kB/s | 121 kB 00:00 2024-01-19T01:12:57.417 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-01-19T01:12:57.479 INFO:teuthology.orchestra.run.smithi079.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-19T01:12:57.492 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): ceph-mgr-rook-17.2.7-268.g6441305d.el9. 140 kB/s | 47 kB 00:00 2024-01-19T01:12:57.517 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 149 kB/s | 15 kB 00:00 2024-01-19T01:12:57.542 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 213 kB/s | 32 kB 00:00 2024-01-19T01:12:57.651 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-google-auth-2.26.0-1.el9.noarc 1.2 MB/s | 202 kB 00:00 2024-01-19T01:12:57.676 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-01-19T01:12:57.710 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-01-19T01:12:57.735 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-01-19T01:12:57.736 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:57.737 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-01-19T01:12:57.737 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:12:57.749 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:12:57.749 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:12:57.763 INFO:teuthology.orchestra.run.smithi079.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-01-19T01:12:57.822 INFO:teuthology.orchestra.run.smithi079.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-01-19T01:12:57.845 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:12:57.846 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:12:58.030 INFO:teuthology.orchestra.run.smithi079.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 793 kB/s | 548 kB 00:00 2024-01-19T01:12:58.032 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:12:58.032 INFO:teuthology.orchestra.run.smithi079.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-01-19T01:12:58.032 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:12:58.042 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:12:58.042 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:12:58.132 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:12:58.132 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:12:58.361 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:12:58.445 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:12:58.532 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-19T01:12:58.587 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-19T01:12:58.600 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-19T01:12:58.641 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-19T01:12:58.648 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-19T01:12:58.678 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-19T01:12:58.745 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-19T01:12:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-19T01:12:58.955 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-19T01:12:59.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-19T01:12:59.158 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-19T01:12:59.242 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-19T01:12:59.297 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-19T01:12:59.345 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T01:12:59.732 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-19T01:12:59.785 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-19T01:12:59.819 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-19T01:12:59.864 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-19T01:12:59.888 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T01:13:00.081 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T01:13:00.082 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/6 2024-01-19T01:13:00.082 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-19T01:13:00.082 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-19T01:13:00.082 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-19T01:13:00.082 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-19T01:13:00.405 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:00.406 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:00.676 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install ceph-fuse 2024-01-19T01:13:00.705 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 13/13 2024-01-19T01:13:00.705 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 1/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-19T01:13:00.706 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-19T01:13:01.078 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-19T01:13:01.078 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:01.078 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:01.078 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:13:01.078 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-19T01:13:01.078 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-19T01:13:01.078 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:01.079 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:01.228 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:33 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:01.343 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-01-19T01:13:01.377 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:01.379 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repository Size 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 811 k 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout:Install 2 Packages 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:01.380 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 889 k 2024-01-19T01:13:01.381 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 2.6 M 2024-01-19T01:13:01.381 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:13:01.508 INFO:teuthology.orchestra.run.smithi079.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 610 kB/s | 79 kB 00:00 2024-01-19T01:13:01.718 INFO:teuthology.orchestra.run.smithi079.stdout:(2/2): ceph-fuse-17.2.7-268.g6441305d.el9.x86_6 2.3 MB/s | 811 kB 00:00 2024-01-19T01:13:01.718 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:01.718 INFO:teuthology.orchestra.run.smithi079.stdout:Total 2.6 MB/s | 889 kB 00:00 2024-01-19T01:13:01.718 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:13:01.738 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:13:01.738 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:13:01.812 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:13:01.812 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:13:01.905 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:34 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:01.990 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:13:02.057 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:02.057 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-01-19T01:13:02.057 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:17.2.7-268.g6441305d.el9 ceph-noarch 121 k 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-01-19T01:13:02.058 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:02.059 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 6.8 M 2024-01-19T01:13:02.059 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-01-19T01:13:02.059 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:13:02.129 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-19T01:13:02.199 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T01:13:02.377 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-19T01:13:02.502 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-01-19T01:13:02.586 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): ceph-mgr-cephadm-17.2.7-268.g6441305d.el 321 kB/s | 121 kB 00:00 2024-01-19T01:13:02.848 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T01:13:02.848 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/2 2024-01-19T01:13:03.078 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 105 kB/s | 58 kB 00:00 2024-01-19T01:13:03.136 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 93 kB/s | 51 kB 00:00 2024-01-19T01:13:03.164 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-19T01:13:03.164 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:03.164 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:13:03.164 INFO:teuthology.orchestra.run.smithi079.stdout: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-19T01:13:03.164 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:03.165 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:03.287 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 603 kB/s | 548 kB 00:00 2024-01-19T01:13:03.290 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:03.290 INFO:teuthology.orchestra.run.smithi152.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-01-19T01:13:03.291 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:13:03.299 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:13:03.300 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:13:03.389 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:13:03.390 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:13:03.448 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados-devel 2024-01-19T01:13:03.688 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:13:03.859 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-19T01:13:03.913 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-19T01:13:04.012 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:36 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:04.162 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 126 k 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:04.163 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 126 k 2024-01-19T01:13:04.164 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 448 k 2024-01-19T01:13:04.164 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:13:04.436 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-19T01:13:04.473 INFO:teuthology.orchestra.run.smithi079.stdout:librados-devel-17.2.7-268.g6441305d.el9.x86_64. 408 kB/s | 126 kB 00:00 2024-01-19T01:13:04.474 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:04.474 INFO:teuthology.orchestra.run.smithi079.stdout:Total 406 kB/s | 126 kB 00:00 2024-01-19T01:13:04.474 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:13:04.480 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:13:04.480 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:13:04.524 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-19T01:13:04.534 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:13:04.535 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:13:04.579 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-19T01:13:04.608 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T01:13:04.688 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:13:04.789 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:05.237 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:05.431 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 6/6 2024-01-19T01:13:05.431 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 1/6 2024-01-19T01:13:05.431 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-19T01:13:05.431 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-19T01:13:05.431 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-19T01:13:05.431 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-19T01:13:05.539 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:05.539 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:05.539 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:13:05.539 INFO:teuthology.orchestra.run.smithi079.stdout: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:05.539 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:05.539 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:05.743 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:17.2.7-268.g6441305d.el9.noarch 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:05.744 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:05.828 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs2 2024-01-19T01:13:05.991 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-01-19T01:13:06.400 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:38 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:06.501 INFO:teuthology.orchestra.run.smithi079.stdout:Package libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:06.545 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:39 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:06.552 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:06.553 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:13:06.553 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:06.632 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install libcephfs-devel 2024-01-19T01:13:06.695 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 811 k 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:06.696 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-01-19T01:13:06.697 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:06.697 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 889 k 2024-01-19T01:13:06.697 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 2.6 M 2024-01-19T01:13:06.697 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:13:06.813 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 681 kB/s | 79 kB 00:00 2024-01-19T01:13:07.129 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-fuse-17.2.7-268.g6441305d.el9.x86_6 1.8 MB/s | 811 kB 00:00 2024-01-19T01:13:07.130 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:07.131 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.0 MB/s | 889 kB 00:00 2024-01-19T01:13:07.131 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:13:07.151 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:13:07.151 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:13:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:39 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:07.225 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:13:07.226 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:13:07.345 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:07.345 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:07.345 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 25 k 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 110 k 2024-01-19T01:13:07.346 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:13:07.407 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:13:07.523 INFO:teuthology.orchestra.run.smithi079.stdout:libcephfs-devel-17.2.7-268.g6441305d.el9.x86_64 141 kB/s | 25 kB 00:00 2024-01-19T01:13:07.524 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:07.524 INFO:teuthology.orchestra.run.smithi079.stdout:Total 140 kB/s | 25 kB 00:00 2024-01-19T01:13:07.524 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:13:07.528 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:13:07.528 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:13:07.529 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:13:07.529 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:13:07.541 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-19T01:13:07.603 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T01:13:07.609 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:13:07.721 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:08.108 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:08.349 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2/2 2024-01-19T01:13:08.349 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/2 2024-01-19T01:13:08.378 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:08.378 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:08.378 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:13:08.378 INFO:teuthology.orchestra.run.smithi079.stdout: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:08.378 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:08.378 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:08.591 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librados2 2024-01-19T01:13:08.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-19T01:13:08.658 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:08.658 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:08.658 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:17.2.7-268.g6441305d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-19T01:13:08.658 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:08.658 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:08.940 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-01-19T01:13:09.154 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:41 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:09.253 INFO:teuthology.orchestra.run.smithi079.stdout:Package librados2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:09.303 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:09.304 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:13:09.304 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:09.389 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install librbd1 2024-01-19T01:13:09.518 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:42 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:09.671 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 126 k 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 126 k 2024-01-19T01:13:09.672 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 448 k 2024-01-19T01:13:09.673 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:13:09.944 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:41 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:09.983 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-17.2.7-268.g6441305d.el9.x86_64. 407 kB/s | 126 kB 00:00 2024-01-19T01:13:09.984 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:09.984 INFO:teuthology.orchestra.run.smithi152.stdout:Total 404 kB/s | 126 kB 00:00 2024-01-19T01:13:09.984 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:13:09.990 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:13:09.990 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:13:10.042 INFO:teuthology.orchestra.run.smithi079.stdout:Package librbd1-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:10.047 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:13:10.047 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:13:10.094 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:10.094 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:13:10.095 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:10.178 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rados 2024-01-19T01:13:10.188 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:13:10.297 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:10.729 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:42 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:10.754 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:10.828 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:10.878 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:10.878 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:13:10.879 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:10.951 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rgw 2024-01-19T01:13:11.050 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:11.050 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:11.050 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:11.050 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:11.050 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:11.050 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:11.285 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-01-19T01:13:11.506 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:43 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:11.604 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:11.654 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:11.655 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:13:11.655 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:11.735 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-cephfs 2024-01-19T01:13:11.854 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:44 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:11.958 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:12.009 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:12.010 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:13:12.010 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:12.084 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-01-19T01:13:12.292 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:44 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:12.442 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:12.443 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:13:12.443 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:12.522 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install python3-rbd 2024-01-19T01:13:12.664 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:45 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:12.816 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:12.817 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:12.817 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-01-19T01:13:12.817 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:12.817 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:13:12.817 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:17.2.7-268.g6441305d.el9 ceph 25 k 2024-01-19T01:13:12.817 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:12.817 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:13:12.818 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:12.818 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-01-19T01:13:12.818 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:12.818 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 25 k 2024-01-19T01:13:12.818 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 110 k 2024-01-19T01:13:12.818 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:13:13.002 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-17.2.7-268.g6441305d.el9.x86_64 135 kB/s | 25 kB 00:00 2024-01-19T01:13:13.003 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:13.003 INFO:teuthology.orchestra.run.smithi152.stdout:Total 135 kB/s | 25 kB 00:00 2024-01-19T01:13:13.003 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:13:13.006 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:13:13.006 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:13:13.008 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:13:13.008 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:13:13.080 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:45 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:13.085 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:13:13.180 INFO:teuthology.orchestra.run.smithi079.stdout:Package python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:13.200 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:13.229 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:13.231 INFO:teuthology.orchestra.run.smithi079.stdout:Nothing to do. 2024-01-19T01:13:13.231 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:13.312 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-fuse 2024-01-19T01:13:13.587 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:13.866 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:13.866 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:13.866 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:13.866 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:13.866 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:13.866 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:13.869 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:45 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:14.020 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:14.021 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:14.021 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-01-19T01:13:14.021 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:14.021 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 81 k 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 81 k 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 222 k 2024-01-19T01:13:14.022 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:13:14.079 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-01-19T01:13:14.235 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-fuse-17.2.7-268.g6441305d.el9.x86_64.rpm 383 kB/s | 81 kB 00:00 2024-01-19T01:13:14.235 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:14.235 INFO:teuthology.orchestra.run.smithi079.stdout:Total 381 kB/s | 81 kB 00:00 2024-01-19T01:13:14.236 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:13:14.242 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:13:14.242 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:13:14.296 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:13:14.296 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:13:14.430 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:13:14.534 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:14.640 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:47 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:14.739 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:14.791 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:14.791 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:13:14.791 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:14.857 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-01-19T01:13:14.924 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:15.200 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:15.200 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:15.200 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:13:15.200 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:15.200 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:15.200 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:15.434 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:48 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:15.447 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-mirror 2024-01-19T01:13:15.541 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:15.595 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:15.596 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:13:15.597 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:15.674 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-01-19T01:13:16.011 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:48 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:16.174 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:16.175 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:16.175 INFO:teuthology.orchestra.run.smithi079.stdout: Package Arch Version Repo Size 2024-01-19T01:13:16.175 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:16.175 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:13:16.175 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T01:13:16.175 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:16.176 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:13:16.176 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:16.176 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-01-19T01:13:16.176 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:16.176 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 2.9 M 2024-01-19T01:13:16.176 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 12 M 2024-01-19T01:13:16.176 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:13:16.258 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:49 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:16.363 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:16.416 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:16.417 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:13:16.417 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:16.488 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-01-19T01:13:16.600 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-mirror-17.2.7-268.g6441305d.el9.x86_64.rpm 7.0 MB/s | 2.9 MB 00:00 2024-01-19T01:13:16.600 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:16.600 INFO:teuthology.orchestra.run.smithi079.stdout:Total 6.9 MB/s | 2.9 MB 00:00 2024-01-19T01:13:16.600 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:13:16.608 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:13:16.608 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:13:16.679 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:13:16.680 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:13:17.046 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:50 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:17.080 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:13:17.146 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:17.157 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:17.181 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:17.182 INFO:teuthology.orchestra.run.smithi079.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:13:17.182 INFO:teuthology.orchestra.run.smithi079.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-19T01:13:17.182 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T01:13:17.182 INFO:teuthology.orchestra.run.smithi079.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T01:13:17.182 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:17.199 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:17.200 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:13:17.200 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:17.269 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-01-19T01:13:17.831 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:50 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:17.932 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:17.982 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:17.983 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:13:17.983 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:18.053 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-01-19T01:13:18.341 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:18.341 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:18.341 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:13:18.341 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:18.341 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:18.341 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:18.595 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum -y install rbd-nbd 2024-01-19T01:13:18.612 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:51 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:18.712 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:17.2.7-268.g6441305d.el9.x86_64 is already installed. 2024-01-19T01:13:18.762 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:18.763 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-01-19T01:13:18.764 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:18.838 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-01-19T01:13:19.163 INFO:teuthology.orchestra.run.smithi079.stdout:Last metadata expiration check: 0:01:51 ago on Fri 19 Jan 2024 01:11:28 AM UTC. 2024-01-19T01:13:19.313 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-01-19T01:13:19.314 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:19.314 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-01-19T01:13:19.314 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:19.314 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-01-19T01:13:19.314 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 164 k 2024-01-19T01:13:19.314 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:19.315 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-01-19T01:13:19.315 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-01-19T01:13:19.315 INFO:teuthology.orchestra.run.smithi079.stdout:Install 1 Package 2024-01-19T01:13:19.315 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:19.315 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 164 k 2024-01-19T01:13:19.315 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 477 k 2024-01-19T01:13:19.315 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-01-19T01:13:19.397 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:52 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:19.546 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:19.547 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:19.547 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-01-19T01:13:19.547 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:19.547 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:13:19.547 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:17.2.7-268.g6441305d.el9 ceph 81 k 2024-01-19T01:13:19.547 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:19.548 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:13:19.548 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:19.548 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-01-19T01:13:19.548 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:19.548 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 81 k 2024-01-19T01:13:19.548 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 222 k 2024-01-19T01:13:19.548 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:13:19.663 INFO:teuthology.orchestra.run.smithi079.stdout:rbd-nbd-17.2.7-268.g6441305d.el9.x86_64.rpm 473 kB/s | 164 kB 00:00 2024-01-19T01:13:19.663 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:19.663 INFO:teuthology.orchestra.run.smithi079.stdout:Total 471 kB/s | 164 kB 00:00 2024-01-19T01:13:19.663 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-01-19T01:13:19.671 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-01-19T01:13:19.672 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-01-19T01:13:19.725 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-01-19T01:13:19.725 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-01-19T01:13:19.855 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-17.2.7-268.g6441305d.el9.x86_64.rpm 264 kB/s | 81 kB 00:00 2024-01-19T01:13:19.856 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:19.856 INFO:teuthology.orchestra.run.smithi152.stdout:Total 263 kB/s | 81 kB 00:00 2024-01-19T01:13:19.857 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:13:19.863 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:13:19.863 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:13:19.878 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-01-19T01:13:19.917 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:13:19.918 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:13:20.009 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:20.057 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:13:20.144 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:20.451 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:20.562 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:20.777 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:20.777 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:20.778 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-01-19T01:13:20.778 INFO:teuthology.orchestra.run.smithi079.stdout: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:20.778 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:13:20.778 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-01-19T01:13:20.858 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:20.858 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:20.858 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:20.858 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:20.858 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:20.858 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:21.007 DEBUG:teuthology.parallel:result is None 2024-01-19T01:13:21.080 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-01-19T01:13:21.654 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:54 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:21.809 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:21.809 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:21.809 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:17.2.7-268.g6441305d.el9 ceph 2.9 M 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-01-19T01:13:21.810 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:21.811 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.9 M 2024-01-19T01:13:21.811 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 12 M 2024-01-19T01:13:21.811 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:13:22.341 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-17.2.7-268.g6441305d.el9.x86_64.rpm 5.5 MB/s | 2.9 MB 00:00 2024-01-19T01:13:22.342 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:22.342 INFO:teuthology.orchestra.run.smithi152.stdout:Total 5.5 MB/s | 2.9 MB 00:00 2024-01-19T01:13:22.342 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:13:22.351 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:13:22.351 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:13:22.425 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:13:22.425 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:13:22.827 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:13:22.906 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:22.931 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:22.931 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T01:13:22.931 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-19T01:13:22.931 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T01:13:22.931 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T01:13:22.931 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:24.031 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:24.031 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:24.031 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:24.031 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:24.032 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:24.032 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:24.261 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-01-19T01:13:24.831 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:57 ago on Fri 19 Jan 2024 01:11:27 AM UTC. 2024-01-19T01:13:24.982 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-01-19T01:13:24.982 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:17.2.7-268.g6441305d.el9 ceph 164 k 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 164 k 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 477 k 2024-01-19T01:13:24.983 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-01-19T01:13:25.194 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-17.2.7-268.g6441305d.el9.x86_64.rpm 779 kB/s | 164 kB 00:00 2024-01-19T01:13:25.195 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:25.195 INFO:teuthology.orchestra.run.smithi152.stdout:Total 774 kB/s | 164 kB 00:00 2024-01-19T01:13:25.195 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-01-19T01:13:25.202 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-01-19T01:13:25.202 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-01-19T01:13:25.257 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-01-19T01:13:25.257 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-01-19T01:13:25.396 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-01-19T01:13:25.527 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:25.948 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:26.292 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 1/1 2024-01-19T01:13:26.292 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:26.292 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-01-19T01:13:26.292 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:17.2.7-268.g6441305d.el9.x86_64 2024-01-19T01:13:26.292 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:13:26.292 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-01-19T01:13:26.583 DEBUG:teuthology.parallel:result is None 2024-01-19T01:13:26.584 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T01:13:26.584 INFO:teuthology.packaging:ref: None 2024-01-19T01:13:26.584 INFO:teuthology.packaging:tag: None 2024-01-19T01:13:26.584 INFO:teuthology.packaging:branch: quincy 2024-01-19T01:13:26.584 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:13:26.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T01:13:26.765 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-19T01:13:26.789 INFO:teuthology.orchestra.run.smithi079.stdout:17.2.7-268.g6441305d.el9 2024-01-19T01:13:26.789 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-268.g6441305d.el9 2024-01-19T01:13:26.789 INFO:teuthology.task.install:The correct ceph version 17.2.7-268.g6441305d is installed. 2024-01-19T01:13:26.791 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T01:13:26.791 INFO:teuthology.packaging:ref: None 2024-01-19T01:13:26.791 INFO:teuthology.packaging:tag: None 2024-01-19T01:13:26.791 INFO:teuthology.packaging:branch: quincy 2024-01-19T01:13:26.791 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:13:26.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-01-19T01:13:26.965 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-19T01:13:26.986 INFO:teuthology.orchestra.run.smithi152.stdout:17.2.7-268.g6441305d.el9 2024-01-19T01:13:26.986 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-268.g6441305d.el9 2024-01-19T01:13:26.986 INFO:teuthology.task.install:The correct ceph version 17.2.7-268.g6441305d is installed. 2024-01-19T01:13:26.988 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-19T01:13:26.988 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:13:26.988 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-19T01:13:27.020 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:13:27.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-19T01:13:27.053 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-19T01:13:27.053 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:13:27.053 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/daemon-helper 2024-01-19T01:13:27.090 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-19T01:13:27.158 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:13:27.158 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-01-19T01:13:27.187 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-19T01:13:27.254 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-19T01:13:27.254 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:13:27.254 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-19T01:13:27.283 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-19T01:13:27.350 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:13:27.350 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-19T01:13:27.377 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-19T01:13:27.444 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-19T01:13:27.444 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:13:27.444 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/usr/bin/stdin-killer 2024-01-19T01:13:27.471 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-19T01:13:27.538 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:13:27.539 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-01-19T01:13:27.565 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-19T01:13:27.633 INFO:teuthology.run_tasks:Running task print... 2024-01-19T01:13:27.645 INFO:teuthology.task.print:**** done install task... 2024-01-19T01:13:27.645 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-19T01:13:27.744 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-19T01:13:27.744 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-01-19T01:13:27.744 INFO:tasks.cephadm:Cluster fsid is f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:13:27.745 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-19T01:13:27.745 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-19T01:13:27.745 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi079': '172.21.15.79', 'mon.smithi152': '172.21.15.152'} 2024-01-19T01:13:27.745 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-19T01:13:27.745 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-01-19T01:13:27.772 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-01-19T01:13:27.800 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-01-19T01:13:27.800 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-19T01:13:28.044 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Jan 19 01:13 /home/ubuntu/cephtest/cephadm 2024-01-19T01:13:28.044 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-19T01:13:28.262 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Jan 19 01:13 /home/ubuntu/cephtest/cephadm 2024-01-19T01:13:28.262 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-19T01:13:28.278 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-19T01:13:28.303 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-01-19T01:13:28.303 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-01-19T01:13:28.322 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-01-19T01:13:29.174 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T01:13:29.273 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T01:13:55.054 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-01-19T01:13:55.055 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)", 2024-01-19T01:13:55.055 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd", 2024-01-19T01:13:55.055 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-01-19T01:13:55.055 INFO:teuthology.orchestra.run.smithi079.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d" 2024-01-19T01:13:55.055 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-01-19T01:13:55.055 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-01-19T01:13:59.012 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-01-19T01:13:59.012 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)", 2024-01-19T01:13:59.012 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd", 2024-01-19T01:13:59.012 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-01-19T01:13:59.012 INFO:teuthology.orchestra.run.smithi152.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d" 2024-01-19T01:13:59.013 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-01-19T01:13:59.013 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-01-19T01:13:59.031 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-01-19T01:13:59.067 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-01-19T01:13:59.097 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-01-19T01:13:59.137 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-01-19T01:13:59.164 INFO:tasks.cephadm:Writing seed config... 2024-01-19T01:13:59.165 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-01-19T01:13:59.165 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-19T01:13:59.166 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-19T01:13:59.167 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-19T01:13:59.168 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-19T01:13:59.169 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:13:59.169 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-19T01:13:59.194 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f3747e56-b667-11ee-95ae-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-19T01:13:59.195 DEBUG:teuthology.orchestra.run.smithi079:mon.smithi079> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@mon.smithi079.service 2024-01-19T01:13:59.237 INFO:tasks.cephadm:Bootstrapping... 2024-01-19T01:13:59.237 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid f3747e56-b667-11ee-95ae-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.79 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-19T01:13:59.442 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-01-19T01:13:59.443 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'f3747e56-b667-11ee-95ae-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.79', '--skip-admin-label'] 2024-01-19T01:13:59.469 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.8.1 2024-01-19T01:13:59.469 INFO:teuthology.orchestra.run.smithi079.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-01-19T01:13:59.469 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying podman|docker is present... 2024-01-19T01:13:59.493 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.8.1 2024-01-19T01:13:59.493 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying lvm2 is present... 2024-01-19T01:13:59.493 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying time synchronization is in place... 2024-01-19T01:13:59.501 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-19T01:13:59.501 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-19T01:13:59.507 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-19T01:13:59.507 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-01-19T01:13:59.513 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-01-19T01:13:59.519 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-01-19T01:13:59.520 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-01-19T01:13:59.520 INFO:teuthology.orchestra.run.smithi079.stdout:Repeating the final host check... 2024-01-19T01:13:59.543 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 4.8.1 2024-01-19T01:13:59.543 INFO:teuthology.orchestra.run.smithi079.stdout:podman (/bin/podman) version 4.8.1 is present 2024-01-19T01:13:59.543 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl is present 2024-01-19T01:13:59.543 INFO:teuthology.orchestra.run.smithi079.stdout:lvcreate is present 2024-01-19T01:13:59.550 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-19T01:13:59.550 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-19T01:13:59.556 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-19T01:13:59.556 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-01-19T01:13:59.562 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-01-19T01:13:59.568 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-01-19T01:13:59.568 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-01-19T01:13:59.569 INFO:teuthology.orchestra.run.smithi079.stdout:Host looks OK 2024-01-19T01:13:59.569 INFO:teuthology.orchestra.run.smithi079.stdout:Cluster fsid: f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:13:59.569 INFO:teuthology.orchestra.run.smithi079.stdout:Acquiring lock 140006468014144 on /run/cephadm/f3747e56-b667-11ee-95ae-87774f69a715.lock 2024-01-19T01:13:59.569 INFO:teuthology.orchestra.run.smithi079.stdout:Lock 140006468014144 acquired on /run/cephadm/f3747e56-b667-11ee-95ae-87774f69a715.lock 2024-01-19T01:13:59.569 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 3300 ... 2024-01-19T01:13:59.569 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 6789 ... 2024-01-19T01:13:59.570 INFO:teuthology.orchestra.run.smithi079.stdout:Base mon IP(s) is [172.21.15.79:3300, 172.21.15.79:6789], mon addrv is [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-01-19T01:13:59.573 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.79 metric 100 2024-01-19T01:13:59.573 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.79 metric 100 2024-01-19T01:13:59.576 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-01-19T01:13:59.576 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-19T01:13:59.576 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1777sec hoplimit 64 pref medium 2024-01-19T01:13:59.579 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-19T01:13:59.579 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-01-19T01:13:59.579 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-19T01:13:59.579 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-19T01:13:59.579 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce87/64 scope link 2024-01-19T01:13:59.579 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-19T01:13:59.580 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-01-19T01:13:59.580 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-01-19T01:13:59.580 INFO:teuthology.orchestra.run.smithi079.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-19T01:13:59.580 INFO:teuthology.orchestra.run.smithi079.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-19T01:13:59.581 INFO:teuthology.orchestra.run.smithi079.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T01:14:00.088 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd 2024-01-19T01:14:00.089 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-01-19T01:14:00.089 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Getting image source signatures 2024-01-19T01:14:00.089 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:6c507150f7e9f028ef978d633e702dab944bbda82bfbc46f6307495cf6dd9fc7 2024-01-19T01:14:00.089 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:aebaf716154c1970d30484d8c64e6de04feea816ffee6c602dc9eb55fbe60ece 2024-01-19T01:14:00.089 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying config sha256:c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd 2024-01-19T01:14:00.089 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Writing manifest to image destination 2024-01-19T01:14:01.001 INFO:teuthology.orchestra.run.smithi079.stdout:ceph: stdout ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable) 2024-01-19T01:14:01.002 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph version: ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable) 2024-01-19T01:14:01.002 INFO:teuthology.orchestra.run.smithi079.stdout:Extracting ceph user uid/gid from container image... 2024-01-19T01:14:01.876 INFO:teuthology.orchestra.run.smithi079.stdout:stat: stdout 167 167 2024-01-19T01:14:01.876 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial keys... 2024-01-19T01:14:02.707 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDazKll2uBmCxAAmgbnfRKq0+eIz+z59Sn9Qg== 2024-01-19T01:14:03.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDbzKllZyPdAhAAODVDCdPb7HX1/83CnAEeMw== 2024-01-19T01:14:04.595 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQDczKllWPcmARAAeQrcZV3fT+Z2+f/GM+O+Mg== 2024-01-19T01:14:04.596 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial monmap... 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool for smithi079 [v2:172.21.15.79:3300,v1:172.21.15.79:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = octopus 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: set fsid to f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:05.495 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-19T01:14:05.496 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:14:05.496 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mon... 2024-01-19T01:14:06.416 INFO:teuthology.orchestra.run.smithi079.stdout:create mon.smithi079 on 2024-01-19T01:14:06.624 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-01-19T01:14:06.766 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-19T01:14:06.910 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f3747e56-b667-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-f3747e56-b667-11ee-95ae-87774f69a715.target. 2024-01-19T01:14:06.910 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-f3747e56-b667-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-f3747e56-b667-11ee-95ae-87774f69a715.target. 2024-01-19T01:14:07.075 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f3747e56-b667-11ee-95ae-87774f69a715@mon.smithi079 2024-01-19T01:14:07.075 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f3747e56-b667-11ee-95ae-87774f69a715@mon.smithi079.service: Unit ceph-f3747e56-b667-11ee-95ae-87774f69a715@mon.smithi079.service not loaded. 2024-01-19T01:14:07.232 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f3747e56-b667-11ee-95ae-87774f69a715.target.wants/ceph-f3747e56-b667-11ee-95ae-87774f69a715@mon.smithi079.service → /etc/systemd/system/ceph-f3747e56-b667-11ee-95ae-87774f69a715@.service. 2024-01-19T01:14:07.718 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:07 smithi079 podman[27981]: 2024-01-19 01:14:07.65657922 +0000 UTC m=+0.297612941 container start e9b85ef40874169e287cd2c0b92115f2736f7c4113b177bb785bbcbada9ade6d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-6441305, GIT_BRANCH=HEAD, org.label-schema.build-date=20240102, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2024-01-19T01:14:07.719 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:07 smithi079 ceph-mon[28000]: mkfs f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:07.758 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T01:14:07.758 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-01-19T01:14:07.765 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T01:14:07.765 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-01-19T01:14:07.765 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-01-19T01:14:07.765 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-01-19T01:14:07.766 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon to start... 2024-01-19T01:14:07.766 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon... 2024-01-19T01:14:08.005 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:07 smithi079 ceph-mon[28000]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-01-19T01:14:08.005 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:07 smithi079 bash[27981]: e9b85ef40874169e287cd2c0b92115f2736f7c4113b177bb785bbcbada9ade6d 2024-01-19T01:14:08.006 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:07 smithi079 systemd[1]: Started Ceph mon.smithi079 for f3747e56-b667-11ee-95ae-87774f69a715. 2024-01-19T01:14:09.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:08 smithi079 ceph-mon[28000]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-01-19T01:14:09.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:08 smithi079 ceph-mon[28000]: monmap e1: 1 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T01:14:09.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:08 smithi079 ceph-mon[28000]: fsmap 2024-01-19T01:14:09.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:08 smithi079 ceph-mon[28000]: osdmap e1: 0 total, 0 up, 0 in 2024-01-19T01:14:09.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:08 smithi079 ceph-mon[28000]: mgrmap e1: no daemons active 2024-01-19T01:14:09.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:08 smithi079 ceph-mon[28000]: from='client.? 172.21.15.79:0/1993235210' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-19T01:14:10.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout cluster: 2024-01-19T01:14:10.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout id: f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:10.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout services: 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi079 (age 0.562958s) 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout data: 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pgs: 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.199+0000 7fd3470e3700 1 Processor -- start 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.199+0000 7fd3470e3700 1 -- start start 2024-01-19T01:14:10.134 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.199+0000 7fd3470e3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd3401015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.199+0000 7fd3470e3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd340101b20 con 0x7fd3401011d0 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.200+0000 7fd344e7f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd3401015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.200+0000 7fd344e7f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd3401015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35558/0 (socket says 172.21.15.79:35558) 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.200+0000 7fd344e7f700 1 -- 172.21.15.79:0/2060701781 learned_addr learned my addr 172.21.15.79:0/2060701781 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.200+0000 7fd344e7f700 1 -- 172.21.15.79:0/2060701781 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340101c60 con 0x7fd3401011d0 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.200+0000 7fd344e7f700 1 --2- 172.21.15.79:0/2060701781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd3401015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd338009e50 tx=0x7fd338009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42f15028d5f0fd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd33f7fe700 1 -- 172.21.15.79:0/2060701781 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3380092e0 con 0x7fd3401011d0 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd33f7fe700 1 -- 172.21.15.79:0/2060701781 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33800ba20 con 0x7fd3401011d0 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd33f7fe700 1 -- 172.21.15.79:0/2060701781 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33800bd30 con 0x7fd3401011d0 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd3470e3700 1 -- 172.21.15.79:0/2060701781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 msgr2=0x7fd3401015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd3470e3700 1 --2- 172.21.15.79:0/2060701781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd3401015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd338009e50 tx=0x7fd338009840 comp rx=0 tx=0).stop 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd3470e3700 1 -- 172.21.15.79:0/2060701781 shutdown_connections 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd3470e3700 1 --2- 172.21.15.79:0/2060701781 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd3401015e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd3470e3700 1 -- 172.21.15.79:0/2060701781 >> 172.21.15.79:0/2060701781 conn(0x7fd3400fc950 msgr2=0x7fd3400fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:10.135 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd3470e3700 1 -- 172.21.15.79:0/2060701781 shutdown_connections 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.201+0000 7fd3470e3700 1 -- 172.21.15.79:0/2060701781 wait complete. 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd3470e3700 1 Processor -- start 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd3470e3700 1 -- start start 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd3470e3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd340110990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd3470e3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd340110ed0 con 0x7fd3401011d0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd344e7f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd340110990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd344e7f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd340110990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35570/0 (socket says 172.21.15.79:35570) 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd344e7f700 1 -- 172.21.15.79:0/1993235210 learned_addr learned my addr 172.21.15.79:0/1993235210 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.202+0000 7fd344e7f700 1 -- 172.21.15.79:0/1993235210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338009ae0 con 0x7fd3401011d0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd344e7f700 1 --2- 172.21.15.79:0/1993235210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd340110990 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd340101f10 tx=0x7fd338017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd33d7fa700 1 -- 172.21.15.79:0/1993235210 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd338012b90 con 0x7fd3401011d0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd33d7fa700 1 -- 172.21.15.79:0/1993235210 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd338012d10 con 0x7fd3401011d0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3401110d0 con 0x7fd3401011d0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd33d7fa700 1 -- 172.21.15.79:0/1993235210 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd33801e400 con 0x7fd3401011d0 2024-01-19T01:14:10.136 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd340111d70 con 0x7fd3401011d0 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd33d7fa700 1 -- 172.21.15.79:0/1993235210 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd338019030 con 0x7fd3401011d0 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.203+0000 7fd33d7fa700 1 -- 172.21.15.79:0/1993235210 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd33801ed00 con 0x7fd3401011d0 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.204+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34005f270 con 0x7fd3401011d0 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.206+0000 7fd33d7fa700 1 -- 172.21.15.79:0/1993235210 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd338017070 con 0x7fd3401011d0 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.277+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd340112780 con 0x7fd3401011d0 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.277+0000 7fd33d7fa700 1 -- 172.21.15.79:0/1993235210 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fd33801e6d0 con 0x7fd3401011d0 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.278+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 msgr2=0x7fd340110990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.278+0000 7fd3470e3700 1 --2- 172.21.15.79:0/1993235210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd340110990 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd340101f10 tx=0x7fd338017000 comp rx=0 tx=0).stop 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.278+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 shutdown_connections 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.278+0000 7fd3470e3700 1 --2- 172.21.15.79:0/1993235210 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd3401011d0 0x7fd340110990 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.278+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 >> 172.21.15.79:0/1993235210 conn(0x7fd3400fc950 msgr2=0x7fd3400fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.278+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 shutdown_connections 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:08.278+0000 7fd3470e3700 1 -- 172.21.15.79:0/1993235210 wait complete. 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:mon is available 2024-01-19T01:14:10.137 INFO:teuthology.orchestra.run.smithi079.stdout:Assimilating anything we can from ceph.conf... 2024-01-19T01:14:13.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:13 smithi079 ceph-mon[28000]: from='client.? 172.21.15.79:0/3858507397' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-19T01:14:13.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:13 smithi079 ceph-mon[28000]: from='client.? 172.21.15.79:0/3858507397' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-19T01:14:16.823 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:16.823 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-01-19T01:14:16.823 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:16.823 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-19T01:14:16.824 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.916+0000 7f688e41c700 1 Processor -- start 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.916+0000 7f688e41c700 1 -- start start 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.916+0000 7f688e41c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.916+0000 7f688e41c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6888102c00 con 0x7f68881022b0 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.916+0000 7f6887fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.916+0000 7f6887fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45894/0 (socket says 172.21.15.79:45894) 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.916+0000 7f6887fff700 1 -- 172.21.15.79:0/657109477 learned_addr learned my addr 172.21.15.79:0/657109477 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f6887fff700 1 -- 172.21.15.79:0/657109477 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6888102d40 con 0x7f68881022b0 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f6887fff700 1 --2- 172.21.15.79:0/657109477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881026c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f68780098e0 tx=0x7f6878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb2af0f193efeee server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f6886ffd700 1 -- 172.21.15.79:0/657109477 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f687800b860 con 0x7f68881022b0 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f6886ffd700 1 -- 172.21.15.79:0/657109477 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687800b9c0 con 0x7f68881022b0 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f6886ffd700 1 -- 172.21.15.79:0/657109477 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f687800bc90 con 0x7f68881022b0 2024-01-19T01:14:16.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f688e41c700 1 -- 172.21.15.79:0/657109477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 msgr2=0x7f68881026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f688e41c700 1 --2- 172.21.15.79:0/657109477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881026c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f68780098e0 tx=0x7f6878009250 comp rx=0 tx=0).stop 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f688e41c700 1 -- 172.21.15.79:0/657109477 shutdown_connections 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f688e41c700 1 --2- 172.21.15.79:0/657109477 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881026c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.917+0000 7f688e41c700 1 -- 172.21.15.79:0/657109477 >> 172.21.15.79:0/657109477 conn(0x7f68880fda10 msgr2=0x7f68880ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f688e41c700 1 -- 172.21.15.79:0/657109477 shutdown_connections 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f688e41c700 1 -- 172.21.15.79:0/657109477 wait complete. 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f688e41c700 1 Processor -- start 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f688e41c700 1 -- start start 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f688e41c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881109b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f688e41c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6888110ef0 con 0x7f68881022b0 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f6887fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f6887fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881109b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45906/0 (socket says 172.21.15.79:45906) 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.918+0000 7f6887fff700 1 -- 172.21.15.79:0/3858507397 learned_addr learned my addr 172.21.15.79:0/3858507397 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f6887fff700 1 -- 172.21.15.79:0/3858507397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68780094f0 con 0x7f68881022b0 2024-01-19T01:14:16.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f6887fff700 1 --2- 172.21.15.79:0/3858507397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881109b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6878003080 tx=0x7f6878016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f687801c5c0 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687801c720 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68881110f0 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f687801ca30 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6888111500 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f687801a450 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.919+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f687802e790 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.920+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688804bb80 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.922+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f687801a890 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.989+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6888103620 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.993+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f687802ee60 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.993+0000 7f6884ff9700 1 -- 172.21.15.79:0/3858507397 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f6878024790 con 0x7f68881022b0 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.994+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 msgr2=0x7f68881109b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:16.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.994+0000 7f688e41c700 1 --2- 172.21.15.79:0/3858507397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881109b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6878003080 tx=0x7f6878016000 comp rx=0 tx=0).stop 2024-01-19T01:14:16.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.994+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 shutdown_connections 2024-01-19T01:14:16.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.994+0000 7f688e41c700 1 --2- 172.21.15.79:0/3858507397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f68881022b0 0x7f68881109b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:16.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.994+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 >> 172.21.15.79:0/3858507397 conn(0x7f68880fda10 msgr2=0x7f6888105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:16.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.994+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 shutdown_connections 2024-01-19T01:14:16.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:12.994+0000 7f688e41c700 1 -- 172.21.15.79:0/3858507397 wait complete. 2024-01-19T01:14:16.828 INFO:teuthology.orchestra.run.smithi079.stdout:Generating new minimal ceph.conf... 2024-01-19T01:14:17.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:17 smithi079 ceph-mon[28000]: from='client.? 172.21.15.79:0/2072637845' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:14:17.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.219+0000 7f36a89af700 1 Processor -- start 2024-01-19T01:14:17.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.219+0000 7f36a89af700 1 -- start start 2024-01-19T01:14:17.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a89af700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:17.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a89af700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a4104310 con 0x7f36a41039c0 2024-01-19T01:14:17.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:17.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45910/0 (socket says 172.21.15.79:45910) 2024-01-19T01:14:17.803 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a259c700 1 -- 172.21.15.79:0/1499088311 learned_addr learned my addr 172.21.15.79:0/1499088311 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a259c700 1 -- 172.21.15.79:0/1499088311 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a4101fb0 con 0x7f36a41039c0 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a259c700 1 --2- 172.21.15.79:0/1499088311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f36980098e0 tx=0x7f3698009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200a473e856e37bb server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a159a700 1 -- 172.21.15.79:0/1499088311 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f369800b860 con 0x7f36a41039c0 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a159a700 1 -- 172.21.15.79:0/1499088311 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f369800be30 con 0x7f36a41039c0 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.220+0000 7f36a159a700 1 -- 172.21.15.79:0/1499088311 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3698010380 con 0x7f36a41039c0 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.221+0000 7f36a89af700 1 -- 172.21.15.79:0/1499088311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 msgr2=0x7f36a4103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.221+0000 7f36a89af700 1 --2- 172.21.15.79:0/1499088311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f36980098e0 tx=0x7f3698009250 comp rx=0 tx=0).stop 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.221+0000 7f36a89af700 1 -- 172.21.15.79:0/1499088311 shutdown_connections 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.221+0000 7f36a89af700 1 --2- 172.21.15.79:0/1499088311 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4103dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.221+0000 7f36a89af700 1 -- 172.21.15.79:0/1499088311 >> 172.21.15.79:0/1499088311 conn(0x7f36a40fda40 msgr2=0x7f36a40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.221+0000 7f36a89af700 1 -- 172.21.15.79:0/1499088311 shutdown_connections 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.221+0000 7f36a89af700 1 -- 172.21.15.79:0/1499088311 wait complete. 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a89af700 1 Processor -- start 2024-01-19T01:14:17.804 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a89af700 1 -- start start 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a89af700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a89af700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a4077190 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45912/0 (socket says 172.21.15.79:45912) 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a259c700 1 -- 172.21.15.79:0/2072637845 learned_addr learned my addr 172.21.15.79:0/2072637845 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a259c700 1 -- 172.21.15.79:0/2072637845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36980094f0 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a259c700 1 --2- 172.21.15.79:0/2072637845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4078d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3698000c00 tx=0x7f3698009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f3692ffd700 1 -- 172.21.15.79:0/2072637845 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3698010a80 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f3692ffd700 1 -- 172.21.15.79:0/2072637845 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f369801a430 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a4077390 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.222+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a40777a0 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.223+0000 7f3692ffd700 1 -- 172.21.15.79:0/2072637845 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36980245b0 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.223+0000 7f3692ffd700 1 -- 172.21.15.79:0/2072637845 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3698024af0 con 0x7f36a41039c0 2024-01-19T01:14:17.805 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.223+0000 7f3692ffd700 1 -- 172.21.15.79:0/2072637845 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f369802eb50 con 0x7f36a41039c0 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.223+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a405f270 con 0x7f36a41039c0 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.225+0000 7f3692ffd700 1 -- 172.21.15.79:0/2072637845 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3698029070 con 0x7f36a41039c0 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.292+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f36a4102d60 con 0x7f36a41039c0 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.292+0000 7f3692ffd700 1 -- 172.21.15.79:0/2072637845 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f3698010be0 con 0x7f36a41039c0 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.293+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 msgr2=0x7f36a4078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.293+0000 7f36a89af700 1 --2- 172.21.15.79:0/2072637845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4078d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3698000c00 tx=0x7f3698009250 comp rx=0 tx=0).stop 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.293+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 shutdown_connections 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.293+0000 7f36a89af700 1 --2- 172.21.15.79:0/2072637845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f36a41039c0 0x7f36a4078d10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.293+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 >> 172.21.15.79:0/2072637845 conn(0x7f36a40fda40 msgr2=0x7f36a40ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.293+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 shutdown_connections 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:17.293+0000 7f36a89af700 1 -- 172.21.15.79:0/2072637845 wait complete. 2024-01-19T01:14:17.806 INFO:teuthology.orchestra.run.smithi079.stdout:Restarting the monitor... 2024-01-19T01:14:18.066 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:17 smithi079 systemd[1]: Stopping Ceph mon.smithi079 for f3747e56-b667-11ee-95ae-87774f69a715... 2024-01-19T01:14:18.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:18 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079[27996]: 2024-01-19T01:14:18.064+0000 7f36f29e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi079 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-19T01:14:18.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:18 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079[27996]: 2024-01-19T01:14:18.064+0000 7f36f29e6700 -1 mon.smithi079@0(leader) e1 *** Got Signal Terminated *** 2024-01-19T01:14:18.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:18 smithi079 podman[28243]: 2024-01-19 01:14:18.279531254 +0000 UTC m=+0.319841249 container died e9b85ef40874169e287cd2c0b92115f2736f7c4113b177bb785bbcbada9ade6d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079, GIT_CLEAN=True, org.label-schema.build-date=20240102, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-6441305, io.buildah.version=1.31.3, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-01-19T01:14:18.790 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:18 smithi079 podman[28243]: 2024-01-19 01:14:18.606125075 +0000 UTC m=+0.646435070 container cleanup e9b85ef40874169e287cd2c0b92115f2736f7c4113b177bb785bbcbada9ade6d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20240102, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-6441305, org.label-schema.name=CentOS Stream 8 Base Image) 2024-01-19T01:14:18.790 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:18 smithi079 bash[28243]: ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079 2024-01-19T01:14:19.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:18 smithi079 podman[28257]: 2024-01-19 01:14:18.790294523 +0000 UTC m=+0.507194550 container remove e9b85ef40874169e287cd2c0b92115f2736f7c4113b177bb785bbcbada9ade6d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-6441305, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20240102, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS) 2024-01-19T01:14:19.042 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 systemd[1]: ceph-f3747e56-b667-11ee-95ae-87774f69a715@mon.smithi079.service: Deactivated successfully. 2024-01-19T01:14:19.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 systemd[1]: Stopped Ceph mon.smithi079 for f3747e56-b667-11ee-95ae-87774f69a715. 2024-01-19T01:14:19.043 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 systemd[1]: Starting Ceph mon.smithi079 for f3747e56-b667-11ee-95ae-87774f69a715... 2024-01-19T01:14:19.297 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 podman[28355]: 2024-01-19 01:14:19.207936506 +0000 UTC m=+0.095333268 container create 0aba73c41ca996abdfdf063885a50227c0601f018accd10afacaac4c0ff16c84 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240102, io.buildah.version=1.31.3, RELEASE=quincy-6441305, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-01-19T01:14:19.297 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 podman[28355]: 2024-01-19 01:14:19.130356186 +0000 UTC m=+0.017752906 image pull c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd quay.ceph.io/ceph-ci/ceph:quincy 2024-01-19T01:14:19.447 INFO:teuthology.orchestra.run.smithi079.stdout:Setting mon public_network to 172.21.0.0/20 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 podman[28355]: 2024-01-19 01:14:19.377116873 +0000 UTC m=+0.264513594 container init 0aba73c41ca996abdfdf063885a50227c0601f018accd10afacaac4c0ff16c84 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-6441305, org.label-schema.build-date=20240102, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 podman[28355]: 2024-01-19 01:14:19.379924004 +0000 UTC m=+0.267320725 container start 0aba73c41ca996abdfdf063885a50227c0601f018accd10afacaac4c0ff16c84 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240102, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, RELEASE=quincy-6441305, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, CEPH_POINT_RELEASE=) 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: set uid:gid to 167:167 (ceph:ceph) 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable), process ceph-mon, pid 2 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: pidfile_write: ignore empty --pid-file 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: load: jerasure load: lrc 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: RocksDB version: 6.15.5 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Compile date Jan 9 2024 2024-01-19T01:14:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: DB SUMMARY 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: DB Session ID: 0UYHYFCQDFO3W92XJZFS 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: CURRENT file: CURRENT 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: IDENTITY file: IDENTITY 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi079/store.db dir, Total Num: 1, files: 000008.sst 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi079/store.db: 000010.log size: 97766 ; 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.error_if_exists: 0 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.create_if_missing: 0 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.paranoid_checks: 1 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.env: 0x562878164a80 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.fs: Posix File System 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.info_log: 0x5628793e7000 2024-01-19T01:14:19.573 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_file_opening_threads: 16 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.statistics: (nil) 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.use_fsync: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_log_file_size: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.keep_log_file_num: 1000 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.recycle_log_file_num: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.allow_fallocate: 1 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.allow_mmap_reads: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.allow_mmap_writes: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.use_direct_reads: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.create_missing_column_families: 0 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.db_log_dir: 2024-01-19T01:14:19.574 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi079/store.db 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.advise_random_on_open: 1 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.db_write_buffer_size: 0 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.write_buffer_manager: 0x56287a15c9f0 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.rate_limiter: (nil) 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-19T01:14:19.575 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.wal_recovery_mode: 2 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.enable_thread_tracking: 0 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.enable_pipelined_write: 0 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.unordered_write: 0 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.row_cache: None 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.wal_filter: None 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-19T01:14:19.576 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.allow_ingest_behind: 0 2024-01-19T01:14:19.577 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.preserve_deletes: 0 2024-01-19T01:14:19.577 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.two_write_queues: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.manual_wal_flush: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.atomic_flush: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.log_readahead_size: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.best_efforts_recovery: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.allow_data_in_errors: 0 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.db_host_id: __hostname__ 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_background_jobs: 2 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_background_compactions: -1 2024-01-19T01:14:19.578 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_subcompactions: 1 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_total_wal_size: 0 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_open_files: -1 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bytes_per_sync: 0 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_readahead_size: 0 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_background_flushes: -1 2024-01-19T01:14:19.579 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Compression algorithms supported: 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kZSTD supported: 0 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kXpressCompression supported: 0 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kLZ4HCCompression supported: 1 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kZlibCompression supported: 1 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kSnappyCompression supported: 1 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kLZ4Compression supported: 1 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: kBZip2Compression supported: 0 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi079/store.db/MANIFEST-000009 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.merge_operator: 2024-01-19T01:14:19.580 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_filter: None 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_filter_factory: None 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.sst_partitioner_factory: None 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56287943cf90) 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_type: 0 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: data_block_index_type: 0 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_shortening: 1 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: hash_index_allow_collision: 1 2024-01-19T01:14:19.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: checksum: 1 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: no_block_cache: 0 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache: 0x56287946f090 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_options: 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: capacity : 536870912 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: num_shard_bits : 4 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: strict_capacity_limit : 0 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_cache_compressed: (nil) 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: persistent_cache: (nil) 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_size: 4096 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_size_deviation: 10 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_restart_interval: 16 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: index_block_restart_interval: 1 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: metadata_block_size: 4096 2024-01-19T01:14:19.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: partition_filters: 0 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: use_delta_encoding: 1 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: whole_key_filtering: 1 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: verify_compression: 0 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: format_version: 4 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: enable_index_compression: 1 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: block_align: 0 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.write_buffer_size: 33554432 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_write_buffer_number: 2 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression: NoCompression 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression: Disabled 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.prefix_extractor: nullptr 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-19T01:14:19.583 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.num_levels: 7 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression_opts.level: 32767 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression_opts.strategy: 0 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-19T01:14:19.584 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compression_opts.enabled: false 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.target_file_size_base: 67108864 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-19T01:14:19.585 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.arena_block_size: 4194304 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.disable_auto_compactions: 0 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-19T01:14:19.586 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.table_properties_collectors: 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.inplace_update_support: 0 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.bloom_locality: 0 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.max_successive_merges: 0 2024-01-19T01:14:19.587 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.paranoid_file_checks: 0 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.force_consistency_checks: 1 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.report_bg_io_stats: 0 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.ttl: 2592000 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.enable_blob_files: false 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.min_blob_size: 0 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.blob_file_size: 268435456 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi079/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-01-19T01:14:19.588 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705626859405740, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705626859406569, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92648, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90743, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13257, "raw_average_key_size": 49, "raw_value_size": 83455, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705626859, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "62370b58-c0d4-44d5-9381-5d64d798144b", "db_session_id": "0UYHYFCQDFO3W92XJZFS"}} 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705626859406878, "job": 1, "event": "recovery_finished"} 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi079/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x562879403180 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: DB pointer 0x5628794d8000 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** DB Stats ** 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T01:14:19.589 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: L0 2/0 92.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.1 0.00 0.00 1 0.001 0 0 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Sum 2/0 92.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.1 0.00 0.00 1 0.001 0 0 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.1 0.00 0.00 1 0.001 0 0 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-01-19T01:14:19.590 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.1 0.00 0.00 1 0.001 0 0 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative compaction: 0.00 GB write, 29.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval compaction: 0.00 GB write, 29.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-01-19T01:14:19.591 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: L0 2/0 92.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.1 0.00 0.00 1 0.001 0 0 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Sum 2/0 92.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.1 0.00 0.00 1 0.001 0 0 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** Compaction Stats [default] ** 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.1 0.00 0.00 1 0.001 0 0 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-19T01:14:19.592 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Cumulative compaction: 0.00 GB write, 29.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: starting mon.smithi079 rank 0 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi079 fsid f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???) e1 preinit fsid f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???).mds e1 new map 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???).mds e1 print_map 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: e1 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:14:19.593 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: legacy client fscid: -1 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout: No filesystems configured 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 bash[28355]: 0aba73c41ca996abdfdf063885a50227c0601f018accd10afacaac4c0ff16c84 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 systemd[1]: Started Ceph mon.smithi079 for f3747e56-b667-11ee-95ae-87774f69a715. 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mon.smithi079 is new leader, mons smithi079 in quorum (ranks 0) 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: monmap e1: 1 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: fsmap 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: osdmap e1: 0 total, 0 up, 0 in 2024-01-19T01:14:19.594 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:19 smithi079 ceph-mon[28374]: mgrmap e1: no daemons active 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.919+0000 7ffb6b6ce700 1 Processor -- start 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.919+0000 7ffb6b6ce700 1 -- start start 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.919+0000 7ffb6b6ce700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb641015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.919+0000 7ffb6b6ce700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb64101b20 con 0x7ffb641011d0 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb6946a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb641015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb6946a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb641015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43980/0 (socket says 172.21.15.79:43980) 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb6946a700 1 -- 172.21.15.79:0/65105177 learned_addr learned my addr 172.21.15.79:0/65105177 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb6946a700 1 -- 172.21.15.79:0/65105177 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb64101c60 con 0x7ffb641011d0 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb6946a700 1 --2- 172.21.15.79:0/65105177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb641015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400bb80 tx=0x7ffb5400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3369530bc54bea server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb63fff700 1 -- 172.21.15.79:0/65105177 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5400dd00 con 0x7ffb641011d0 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb63fff700 1 -- 172.21.15.79:0/65105177 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ffb5400de60 con 0x7ffb641011d0 2024-01-19T01:14:20.467 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.920+0000 7ffb63fff700 1 -- 172.21.15.79:0/65105177 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb54018840 con 0x7ffb641011d0 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.921+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/65105177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 msgr2=0x7ffb641015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.921+0000 7ffb6b6ce700 1 --2- 172.21.15.79:0/65105177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb641015e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400bb80 tx=0x7ffb5400b4f0 comp rx=0 tx=0).stop 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.921+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/65105177 shutdown_connections 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.921+0000 7ffb6b6ce700 1 --2- 172.21.15.79:0/65105177 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb641015e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.921+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/65105177 >> 172.21.15.79:0/65105177 conn(0x7ffb640fc970 msgr2=0x7ffb640fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.921+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/65105177 shutdown_connections 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.921+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/65105177 wait complete. 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6b6ce700 1 Processor -- start 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6b6ce700 1 -- start start 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6b6ce700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb6410ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6b6ce700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6410f440 con 0x7ffb641011d0 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6946a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb6410ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6946a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb6410ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:43988/0 (socket says 172.21.15.79:43988) 2024-01-19T01:14:20.468 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6946a700 1 -- 172.21.15.79:0/3051102846 learned_addr learned my addr 172.21.15.79:0/3051102846 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6946a700 1 -- 172.21.15.79:0/3051102846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5400b790 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6946a700 1 --2- 172.21.15.79:0/3051102846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb6410ef00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb54006d40 tx=0x7ffb5400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb61ffb700 1 -- 172.21.15.79:0/3051102846 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb54010a00 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb61ffb700 1 -- 172.21.15.79:0/3051102846 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ffb5401d050 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6410f640 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.922+0000 7ffb61ffb700 1 -- 172.21.15.79:0/3051102846 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb54022590 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.923+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6410fa50 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.923+0000 7ffb61ffb700 1 -- 172.21.15.79:0/3051102846 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ffb54022740 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.923+0000 7ffb61ffb700 1 -- 172.21.15.79:0/3051102846 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ffb54014070 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.924+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6404bb80 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.926+0000 7ffb61ffb700 1 -- 172.21.15.79:0/3051102846 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ffb54010b60 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.992+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ffb641087e0 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb61ffb700 1 -- 172.21.15.79:0/3051102846 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ffb54022b80 con 0x7ffb641011d0 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 msgr2=0x7ffb6410ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:20.469 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb6b6ce700 1 --2- 172.21.15.79:0/3051102846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb6410ef00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb54006d40 tx=0x7ffb5400b4f0 comp rx=0 tx=0).stop 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 shutdown_connections 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb6b6ce700 1 --2- 172.21.15.79:0/3051102846 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ffb641011d0 0x7ffb6410ef00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 >> 172.21.15.79:0/3051102846 conn(0x7ffb640fc970 msgr2=0x7ffb64105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 shutdown_connections 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:19.997+0000 7ffb6b6ce700 1 -- 172.21.15.79:0/3051102846 wait complete. 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote config to /etc/ceph/ceph.conf 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mgr... 2024-01-19T01:14:20.470 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 9283 ... 2024-01-19T01:14:20.641 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f3747e56-b667-11ee-95ae-87774f69a715@mgr.smithi079.miqczk 2024-01-19T01:14:20.641 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f3747e56-b667-11ee-95ae-87774f69a715@mgr.smithi079.miqczk.service: Unit ceph-f3747e56-b667-11ee-95ae-87774f69a715@mgr.smithi079.miqczk.service not loaded. 2024-01-19T01:14:20.791 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f3747e56-b667-11ee-95ae-87774f69a715.target.wants/ceph-f3747e56-b667-11ee-95ae-87774f69a715@mgr.smithi079.miqczk.service → /etc/systemd/system/ceph-f3747e56-b667-11ee-95ae-87774f69a715@.service. 2024-01-19T01:14:21.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:20 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3051102846' entity='client.admin' 2024-01-19T01:14:21.309 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T01:14:21.310 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-01-19T01:14:21.317 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T01:14:21.317 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-01-19T01:14:21.317 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-01-19T01:14:21.317 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-01-19T01:14:21.325 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T01:14:21.325 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-01-19T01:14:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T01:14:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-01-19T01:14:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-01-19T01:14:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-19T01:14:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr to start... 2024-01-19T01:14:21.333 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr... 2024-01-19T01:14:22.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:22 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2180459396' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "f3747e56-b667-11ee-95ae-87774f69a715", 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-19T01:14:26.429 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-19T01:14:26.430 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-19T01:14:26.438 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-19T01:14:26.438 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-19T01:14:26.438 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-19T01:14:26.438 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:26.439 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-01-19T01:14:07.716537+0000", 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:26.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.497+0000 7fdf4686a700 1 Processor -- start 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.497+0000 7fdf4686a700 1 -- start start 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.498+0000 7fdf4686a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.498+0000 7fdf4686a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf40103100 con 0x7fdf401027b0 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.498+0000 7fdf3ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.498+0000 7fdf3ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44010/0 (socket says 172.21.15.79:44010) 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.498+0000 7fdf3ffff700 1 -- 172.21.15.79:0/4111406005 learned_addr learned my addr 172.21.15.79:0/4111406005 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.498+0000 7fdf3ffff700 1 -- 172.21.15.79:0/4111406005 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf40103240 con 0x7fdf401027b0 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.498+0000 7fdf3ffff700 1 --2- 172.21.15.79:0/4111406005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40102bc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdf300098e0 tx=0x7fdf30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7be9bdea6dffbb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf3effd700 1 -- 172.21.15.79:0/4111406005 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf3000b860 con 0x7fdf401027b0 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf3effd700 1 -- 172.21.15.79:0/4111406005 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdf3000be30 con 0x7fdf401027b0 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf3effd700 1 -- 172.21.15.79:0/4111406005 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf30010380 con 0x7fdf401027b0 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf4686a700 1 -- 172.21.15.79:0/4111406005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 msgr2=0x7fdf40102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:26.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf4686a700 1 --2- 172.21.15.79:0/4111406005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40102bc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdf300098e0 tx=0x7fdf30009250 comp rx=0 tx=0).stop 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf4686a700 1 -- 172.21.15.79:0/4111406005 shutdown_connections 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf4686a700 1 --2- 172.21.15.79:0/4111406005 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40102bc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf4686a700 1 -- 172.21.15.79:0/4111406005 >> 172.21.15.79:0/4111406005 conn(0x7fdf400fc0b0 msgr2=0x7fdf400fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf4686a700 1 -- 172.21.15.79:0/4111406005 shutdown_connections 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.499+0000 7fdf4686a700 1 -- 172.21.15.79:0/4111406005 wait complete. 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.500+0000 7fdf4686a700 1 Processor -- start 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.500+0000 7fdf4686a700 1 -- start start 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.500+0000 7fdf4686a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.500+0000 7fdf4686a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf40105a10 con 0x7fdf401027b0 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.500+0000 7fdf3ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.500+0000 7fdf3ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40107590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:44012/0 (socket says 172.21.15.79:44012) 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.500+0000 7fdf3ffff700 1 -- 172.21.15.79:0/2180459396 learned_addr learned my addr 172.21.15.79:0/2180459396 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf3ffff700 1 -- 172.21.15.79:0/2180459396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf300094f0 con 0x7fdf401027b0 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf3ffff700 1 --2- 172.21.15.79:0/2180459396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40107590 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf30006b00 tx=0x7fdf30009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:26.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf3cff9700 1 -- 172.21.15.79:0/2180459396 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf300109b0 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf40105c10 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf3cff9700 1 -- 172.21.15.79:0/2180459396 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdf30024440 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf3cff9700 1 -- 172.21.15.79:0/2180459396 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf3001a5f0 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf40106020 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf3cff9700 1 -- 172.21.15.79:0/2180459396 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fdf3001ab30 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.501+0000 7fdf3cff9700 1 -- 172.21.15.79:0/2180459396 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdf3002db90 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.502+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4004bb80 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.506+0000 7fdf3cff9700 1 -- 172.21.15.79:0/2180459396 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fdf30006d70 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.578+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdf40102100 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.579+0000 7fdf3cff9700 1 -- 172.21.15.79:0/2180459396 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fdf3002ddf0 con 0x7fdf401027b0 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.580+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 msgr2=0x7fdf40107590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.580+0000 7fdf4686a700 1 --2- 172.21.15.79:0/2180459396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40107590 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf30006b00 tx=0x7fdf30009250 comp rx=0 tx=0).stop 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.580+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 shutdown_connections 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.580+0000 7fdf4686a700 1 --2- 172.21.15.79:0/2180459396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf401027b0 0x7fdf40107590 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:26.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.580+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 >> 172.21.15.79:0/2180459396 conn(0x7fdf400fc0b0 msgr2=0x7fdf40100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:26.444 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.580+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 shutdown_connections 2024-01-19T01:14:26.444 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:22.580+0000 7fdf4686a700 1 -- 172.21.15.79:0/2180459396 wait complete. 2024-01-19T01:14:26.444 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (1/15)... 2024-01-19T01:14:27.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: Activating manager daemon smithi079.miqczk 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: mgrmap e2: smithi079.miqczk(active, starting, since 0.00296983s) 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr metadata", "who": "smithi079.miqczk", "id": "smithi079.miqczk"}]: dispatch 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: Manager daemon smithi079.miqczk is now available 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/mirror_snapshot_schedule"}]: dispatch 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/trash_purge_schedule"}]: dispatch 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:27.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:27 smithi079 ceph-mon[28374]: from='mgr.14100 172.21.15.79:0/3379457073' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:29.726 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:29.726 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-01-19T01:14:29.726 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "f3747e56-b667-11ee-95ae-87774f69a715", 2024-01-19T01:14:29.726 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-01-19T01:14:29.726 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-19T01:14:29.726 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-19T01:14:29.726 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "smithi079" 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:29.727 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-19T01:14:29.728 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-19T01:14:29.729 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-01-19T01:14:07.716537+0000", 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.874+0000 7f1a94c7b700 1 Processor -- start 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.874+0000 7f1a94c7b700 1 -- start start 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.874+0000 7f1a94c7b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a901015d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.874+0000 7f1a94c7b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90101b10 con 0x7f1a901011c0 2024-01-19T01:14:29.730 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a901015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a901015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46366/0 (socket says 172.21.15.79:46366) 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8e59c700 1 -- 172.21.15.79:0/2465735909 learned_addr learned my addr 172.21.15.79:0/2465735909 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8e59c700 1 -- 172.21.15.79:0/2465735909 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a90101c50 con 0x7f1a901011c0 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8e59c700 1 --2- 172.21.15.79:0/2465735909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a901015d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a840098e0 tx=0x7f1a84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18964a718389f42e server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8d59a700 1 -- 172.21.15.79:0/2465735909 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8400b860 con 0x7f1a901011c0 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8d59a700 1 -- 172.21.15.79:0/2465735909 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1a8400be30 con 0x7f1a901011c0 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.875+0000 7f1a8d59a700 1 -- 172.21.15.79:0/2465735909 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a84010380 con 0x7f1a901011c0 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 -- 172.21.15.79:0/2465735909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 msgr2=0x7f1a901015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 --2- 172.21.15.79:0/2465735909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a901015d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a840098e0 tx=0x7f1a84009250 comp rx=0 tx=0).stop 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 -- 172.21.15.79:0/2465735909 shutdown_connections 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 --2- 172.21.15.79:0/2465735909 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a901015d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 -- 172.21.15.79:0/2465735909 >> 172.21.15.79:0/2465735909 conn(0x7f1a900fc980 msgr2=0x7f1a900fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 -- 172.21.15.79:0/2465735909 shutdown_connections 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 -- 172.21.15.79:0/2465735909 wait complete. 2024-01-19T01:14:29.731 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 Processor -- start 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.876+0000 7f1a94c7b700 1 -- start start 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a94c7b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a90113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a94c7b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a90076bf0 con 0x7f1a901011c0 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a8e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a90113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a8e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a90113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46368/0 (socket says 172.21.15.79:46368) 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a8e59c700 1 -- 172.21.15.79:0/932386295 learned_addr learned my addr 172.21.15.79:0/932386295 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a8e59c700 1 -- 172.21.15.79:0/932386295 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a840094f0 con 0x7f1a901011c0 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a8e59c700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a90113e60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1a84006b00 tx=0x7f1a84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a7effd700 1 -- 172.21.15.79:0/932386295 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a840109b0 con 0x7f1a901011c0 2024-01-19T01:14:29.732 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a7effd700 1 -- 172.21.15.79:0/932386295 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1a84024440 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a90076df0 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a7effd700 1 -- 172.21.15.79:0/932386295 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8401a5f0 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.877+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a90114560 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.878+0000 7f1a7effd700 1 -- 172.21.15.79:0/932386295 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 3) v1 ==== 43164+0+0 (secure 0 0 0) 0x7f1a8401a8b0 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.878+0000 7f1a7effd700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7f1a78036470 0x7f1a78038920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.878+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9010a040 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.878+0000 7f1a7effd700 1 -- 172.21.15.79:0/932386295 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1a8404a2e0 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.878+0000 7f1a8dd9b700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7f1a78036470 0x7f1a78038920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.879+0000 7f1a8dd9b700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7f1a78036470 0x7f1a78038920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009990 tx=0x7f1a80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:28.883+0000 7f1a7effd700 1 -- 172.21.15.79:0/932386295 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1a84022340 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.138+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1a9004bb80 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.138+0000 7f1a7effd700 1 -- 172.21.15.79:0/932386295 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f1a84018b40 con 0x7f1a901011c0 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7f1a78036470 msgr2=0x7f1a78038920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:29.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7f1a78036470 0x7f1a78038920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1a80009990 tx=0x7f1a80006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 msgr2=0x7f1a90113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a90113e60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1a84006b00 tx=0x7f1a84009250 comp rx=0 tx=0).stop 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 shutdown_connections 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7f1a78036470 0x7f1a78038920 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 --2- 172.21.15.79:0/932386295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1a901011c0 0x7f1a90113e60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 >> 172.21.15.79:0/932386295 conn(0x7f1a900fc980 msgr2=0x7f1a900fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 shutdown_connections 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:29.140+0000 7f1a94c7b700 1 -- 172.21.15.79:0/932386295 wait complete. 2024-01-19T01:14:29.734 INFO:teuthology.orchestra.run.smithi079.stdout:mgr is available 2024-01-19T01:14:29.844 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:29 smithi079 ceph-mon[28374]: mgrmap e3: smithi079.miqczk(active, since 1.00636s) 2024-01-19T01:14:29.844 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:29 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/932386295' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-19T01:14:30.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:30.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-01-19T01:14:30.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:30.871 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.134+0000 7fe765449700 1 Processor -- start 2024-01-19T01:14:30.872 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.134+0000 7fe765449700 1 -- start start 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.134+0000 7fe765449700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.134+0000 7fe765449700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601053e0 con 0x7fe760076c60 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46376/0 (socket says 172.21.15.79:46376) 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75effd700 1 -- 172.21.15.79:0/143491996 learned_addr learned my addr 172.21.15.79:0/143491996 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75effd700 1 -- 172.21.15.79:0/143491996 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760105520 con 0x7fe760076c60 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75effd700 1 --2- 172.21.15.79:0/143491996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760104ea0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe7500098e0 tx=0x7fe750009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fd81e23cc0dc79 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75e7fc700 1 -- 172.21.15.79:0/143491996 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe75000b860 con 0x7fe760076c60 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75e7fc700 1 -- 172.21.15.79:0/143491996 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe75000be30 con 0x7fe760076c60 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.135+0000 7fe75e7fc700 1 -- 172.21.15.79:0/143491996 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe750010380 con 0x7fe760076c60 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 -- 172.21.15.79:0/143491996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 msgr2=0x7fe760104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 --2- 172.21.15.79:0/143491996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760104ea0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe7500098e0 tx=0x7fe750009250 comp rx=0 tx=0).stop 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 -- 172.21.15.79:0/143491996 shutdown_connections 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 --2- 172.21.15.79:0/143491996 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760104ea0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:30.873 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 -- 172.21.15.79:0/143491996 >> 172.21.15.79:0/143491996 conn(0x7fe7600fc950 msgr2=0x7fe7600fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 -- 172.21.15.79:0/143491996 shutdown_connections 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 -- 172.21.15.79:0/143491996 wait complete. 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.136+0000 7fe765449700 1 Processor -- start 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe765449700 1 -- start start 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe765449700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe765449700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760113890 con 0x7fe760076c60 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe75effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe75effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46390/0 (socket says 172.21.15.79:46390) 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe75effd700 1 -- 172.21.15.79:0/2344970138 learned_addr learned my addr 172.21.15.79:0/2344970138 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe75effd700 1 -- 172.21.15.79:0/2344970138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500094f0 con 0x7fe760076c60 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe75effd700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760113350 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe750006b00 tx=0x7fe750009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe757fff700 1 -- 172.21.15.79:0/2344970138 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe750010a80 con 0x7fe760076c60 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe757fff700 1 -- 172.21.15.79:0/2344970138 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe75001a430 con 0x7fe760076c60 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.137+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe760113a90 con 0x7fe760076c60 2024-01-19T01:14:30.874 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.138+0000 7fe757fff700 1 -- 172.21.15.79:0/2344970138 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7500245b0 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.138+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe760113ea0 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.138+0000 7fe757fff700 1 -- 172.21.15.79:0/2344970138 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 3) v1 ==== 43164+0+0 (secure 0 0 0) 0x7fe750024830 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.139+0000 7fe757fff700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fe74c036470 0x7fe74c038920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.139+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76004bb80 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.139+0000 7fe757fff700 1 -- 172.21.15.79:0/2344970138 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe75004a890 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.139+0000 7fe7567fc700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fe74c036470 0x7fe74c038920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.139+0000 7fe7567fc700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fe74c036470 0x7fe74c038920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe748009990 tx=0x7fe748006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.143+0000 7fe757fff700 1 -- 172.21.15.79:0/2344970138 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe75002cd10 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.324+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe760101840 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.325+0000 7fe757fff700 1 -- 172.21.15.79:0/2344970138 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe75002cb30 con 0x7fe760076c60 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fe74c036470 msgr2=0x7fe74c038920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fe74c036470 0x7fe74c038920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe748009990 tx=0x7fe748006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 msgr2=0x7fe760113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:30.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760113350 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe750006b00 tx=0x7fe750009250 comp rx=0 tx=0).stop 2024-01-19T01:14:30.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 shutdown_connections 2024-01-19T01:14:30.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fe74c036470 0x7fe74c038920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:30.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 --2- 172.21.15.79:0/2344970138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe760076c60 0x7fe760113350 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:30.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.327+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 >> 172.21.15.79:0/2344970138 conn(0x7fe7600fc950 msgr2=0x7fe760103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:30.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.328+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 shutdown_connections 2024-01-19T01:14:30.876 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:30.328+0000 7fe765449700 1 -- 172.21.15.79:0/2344970138 wait complete. 2024-01-19T01:14:30.876 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling cephadm module... 2024-01-19T01:14:30.989 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:30 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2344970138' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-19T01:14:31.983 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:31 smithi079 ceph-mon[28374]: mgrmap e4: smithi079.miqczk(active, since 3s) 2024-01-19T01:14:31.983 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:31 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/4053278685' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-19T01:14:32.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.267+0000 7ff140b74700 1 Processor -- start 2024-01-19T01:14:32.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.267+0000 7ff140b74700 1 -- start start 2024-01-19T01:14:32.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.267+0000 7ff140b74700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:32.333 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.267+0000 7ff140b74700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c104240 con 0x7ff13c1038f0 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.267+0000 7ff13a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.267+0000 7ff13a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46400/0 (socket says 172.21.15.79:46400) 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.267+0000 7ff13a59c700 1 -- 172.21.15.79:0/3726365488 learned_addr learned my addr 172.21.15.79:0/3726365488 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff13a59c700 1 -- 172.21.15.79:0/3726365488 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c104380 con 0x7ff13c1038f0 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff13a59c700 1 --2- 172.21.15.79:0/3726365488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103d00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0098e0 tx=0x7ff12c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f963fb7840afbd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff139d9b700 1 -- 172.21.15.79:0/3726365488 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c00b860 con 0x7ff13c1038f0 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff139d9b700 1 -- 172.21.15.79:0/3726365488 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff12c00be30 con 0x7ff13c1038f0 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff139d9b700 1 -- 172.21.15.79:0/3726365488 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c010380 con 0x7ff13c1038f0 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff140b74700 1 -- 172.21.15.79:0/3726365488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 msgr2=0x7ff13c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff140b74700 1 --2- 172.21.15.79:0/3726365488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103d00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0098e0 tx=0x7ff12c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff140b74700 1 -- 172.21.15.79:0/3726365488 shutdown_connections 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff140b74700 1 --2- 172.21.15.79:0/3726365488 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103d00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.268+0000 7ff140b74700 1 -- 172.21.15.79:0/3726365488 >> 172.21.15.79:0/3726365488 conn(0x7ff13c0fc0b0 msgr2=0x7ff13c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff140b74700 1 -- 172.21.15.79:0/3726365488 shutdown_connections 2024-01-19T01:14:32.334 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff140b74700 1 -- 172.21.15.79:0/3726365488 wait complete. 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff140b74700 1 Processor -- start 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff140b74700 1 -- start start 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff140b74700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff140b74700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c1015b0 con 0x7ff13c1038f0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff13a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff13a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46410/0 (socket says 172.21.15.79:46410) 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.269+0000 7ff13a59c700 1 -- 172.21.15.79:0/4053278685 learned_addr learned my addr 172.21.15.79:0/4053278685 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.270+0000 7ff13a59c700 1 -- 172.21.15.79:0/4053278685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0094f0 con 0x7ff13c1038f0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.270+0000 7ff13a59c700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff12c000c00 tx=0x7ff12c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.270+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c010a80 con 0x7ff13c1038f0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.270+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff12c01a430 con 0x7ff13c1038f0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.270+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c1017b0 con 0x7ff13c1038f0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.270+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff12c0245b0 con 0x7ff13c1038f0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.270+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c101bc0 con 0x7ff13c1038f0 2024-01-19T01:14:32.335 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.271+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7ff12c0247f0 con 0x7ff13c1038f0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.271+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c04bb80 con 0x7ff13c1038f0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.271+0000 7ff1337fe700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7ff1280365a0 0x7ff128038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.271+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff12c04ab00 con 0x7ff13c1038f0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.271+0000 7ff131d9b700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7ff1280365a0 0x7ff128038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.272+0000 7ff131d9b700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7ff1280365a0 0x7ff128038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff124009990 tx=0x7ff124006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.276+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff12c04ce60 con 0x7ff13c1038f0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.510+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff13c100dd0 con 0x7ff13c1038f0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.689+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7ff12c021d50 con 0x7ff13c1038f0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.690+0000 7ff1337fe700 1 -- 172.21.15.79:0/4053278685 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff12c023dd0 con 0x7ff13c1038f0 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7ff1280365a0 msgr2=0x7ff128038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7ff1280365a0 0x7ff128038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff124009990 tx=0x7ff124006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 msgr2=0x7ff13c103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103130 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff12c000c00 tx=0x7ff12c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:32.336 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 shutdown_connections 2024-01-19T01:14:32.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7ff1280365a0 0x7ff128038a50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:32.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 --2- 172.21.15.79:0/4053278685 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7ff13c1038f0 0x7ff13c103130 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:32.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 >> 172.21.15.79:0/4053278685 conn(0x7ff13c0fc0b0 msgr2=0x7ff13c0fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:32.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 shutdown_connections 2024-01-19T01:14:32.337 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:31.694+0000 7ff140b74700 1 -- 172.21.15.79:0/4053278685 wait complete. 2024-01-19T01:14:32.938 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:32 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/4053278685' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-19T01:14:32.938 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:32 smithi079 ceph-mon[28374]: mgrmap e5: smithi079.miqczk(active, since 4s) 2024-01-19T01:14:33.632 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "smithi079.miqczk", 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.732+0000 7fbee7701700 1 Processor -- start 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.732+0000 7fbee7701700 1 -- start start 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.732+0000 7fbee7701700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.732+0000 7fbee7701700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0105420 con 0x7fbee0070fc0 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.732+0000 7fbee549d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.732+0000 7fbee549d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46432/0 (socket says 172.21.15.79:46432) 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.732+0000 7fbee549d700 1 -- 172.21.15.79:0/3086620891 learned_addr learned my addr 172.21.15.79:0/3086620891 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:33.633 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.733+0000 7fbee549d700 1 -- 172.21.15.79:0/3086620891 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee0105560 con 0x7fbee0070fc0 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.733+0000 7fbee549d700 1 --2- 172.21.15.79:0/3086620891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0104ee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0098e0 tx=0x7fbedc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b727f3f66421e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.733+0000 7fbed7fff700 1 -- 172.21.15.79:0/3086620891 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc00b860 con 0x7fbee0070fc0 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.733+0000 7fbed7fff700 1 -- 172.21.15.79:0/3086620891 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbedc00be30 con 0x7fbee0070fc0 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.733+0000 7fbed7fff700 1 -- 172.21.15.79:0/3086620891 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc010380 con 0x7fbee0070fc0 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.733+0000 7fbee7701700 1 -- 172.21.15.79:0/3086620891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 msgr2=0x7fbee0104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.733+0000 7fbee7701700 1 --2- 172.21.15.79:0/3086620891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0104ee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0098e0 tx=0x7fbedc009250 comp rx=0 tx=0).stop 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 -- 172.21.15.79:0/3086620891 shutdown_connections 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 --2- 172.21.15.79:0/3086620891 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0104ee0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 -- 172.21.15.79:0/3086620891 >> 172.21.15.79:0/3086620891 conn(0x7fbee00fc970 msgr2=0x7fbee00fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:33.634 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 -- 172.21.15.79:0/3086620891 shutdown_connections 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 -- 172.21.15.79:0/3086620891 wait complete. 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 Processor -- start 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 -- start start 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0111180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee7701700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01116c0 con 0x7fbee0070fc0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee549d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0111180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee549d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0111180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46442/0 (socket says 172.21.15.79:46442) 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.734+0000 7fbee549d700 1 -- 172.21.15.79:0/1082107792 learned_addr learned my addr 172.21.15.79:0/1082107792 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.735+0000 7fbee549d700 1 -- 172.21.15.79:0/1082107792 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc0094f0 con 0x7fbee0070fc0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.735+0000 7fbee549d700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0111180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006b00 tx=0x7fbedc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.735+0000 7fbed5ffb700 1 -- 172.21.15.79:0/1082107792 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc010a80 con 0x7fbee0070fc0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.735+0000 7fbed5ffb700 1 -- 172.21.15.79:0/1082107792 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbedc01a430 con 0x7fbee0070fc0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.735+0000 7fbed5ffb700 1 -- 172.21.15.79:0/1082107792 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbedc0245b0 con 0x7fbee0070fc0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.735+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee01118c0 con 0x7fbee0070fc0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.735+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee0111cd0 con 0x7fbee0070fc0 2024-01-19T01:14:33.635 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.736+0000 7fbed5ffb700 1 -- 172.21.15.79:0/1082107792 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fbedc010be0 con 0x7fbee0070fc0 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.736+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee004bb80 con 0x7fbee0070fc0 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.736+0000 7fbed5ffb700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 0x7fbecc038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.736+0000 7fbed5ffb700 1 -- 172.21.15.79:0/1082107792 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbedc04aae0 con 0x7fbee0070fc0 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.736+0000 7fbee4c9c700 1 -- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 msgr2=0x7fbecc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/430351410 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.736+0000 7fbee4c9c700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 0x7fbecc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.741+0000 7fbed5ffb700 1 -- 172.21.15.79:0/1082107792 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbedc01a5a0 con 0x7fbee0070fc0 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.937+0000 7fbee4c9c700 1 -- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 msgr2=0x7fbecc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/430351410 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.937+0000 7fbee4c9c700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 0x7fbecc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.977+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbee00008d0 con 0x7fbee0070fc0 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.977+0000 7fbed5ffb700 1 -- 172.21.15.79:0/1082107792 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fbedc023b80 con 0x7fbee0070fc0 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 msgr2=0x7fbecc038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 0x7fbecc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:33.636 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 msgr2=0x7fbee0111180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0111180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006b00 tx=0x7fbedc009250 comp rx=0 tx=0).stop 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 shutdown_connections 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fbecc036640 0x7fbecc038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 --2- 172.21.15.79:0/1082107792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbee0070fc0 0x7fbee0111180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.979+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 >> 172.21.15.79:0/1082107792 conn(0x7fbee00fc970 msgr2=0x7fbee00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.980+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 shutdown_connections 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:32.980+0000 7fbee7701700 1 -- 172.21.15.79:0/1082107792 wait complete. 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-01-19T01:14:33.637 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 5... 2024-01-19T01:14:33.775 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:33 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1082107792' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-19T01:14:38.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: Active manager daemon smithi079.miqczk restarted 2024-01-19T01:14:38.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: Activating manager daemon smithi079.miqczk 2024-01-19T01:14:38.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: osdmap e2: 0 total, 0 up, 0 in 2024-01-19T01:14:38.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: mgrmap e6: smithi079.miqczk(active, starting, since 0.0033634s) 2024-01-19T01:14:38.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:14:38.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr metadata", "who": "smithi079.miqczk", "id": "smithi079.miqczk"}]: dispatch 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: Manager daemon smithi079.miqczk is now available 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:14:38.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:38 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:14:39.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:39 smithi079 ceph-mon[28374]: Found migration_current of "None". Setting to last migration. 2024-01-19T01:14:39.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:39 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/mirror_snapshot_schedule"}]: dispatch 2024-01-19T01:14:39.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:39 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/trash_purge_schedule"}]: dispatch 2024-01-19T01:14:39.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:39 smithi079 ceph-mon[28374]: [19/Jan/2024:01:14:38] ENGINE Bus STARTING 2024-01-19T01:14:39.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:39 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:39.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:39 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:14:39.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:39 smithi079 ceph-mon[28374]: mgrmap e7: smithi079.miqczk(active, since 1.00572s) 2024-01-19T01:14:39.875 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-01-19T01:14:39.884 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-01-19T01:14:39.884 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-19T01:14:39.884 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-01-19T01:14:39.884 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e2ca1700 1 Processor -- start 2024-01-19T01:14:39.884 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e2ca1700 1 -- start start 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e2ca1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc1026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e2ca1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc102c00 con 0x7fd4dc1022b0 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e0a3d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc1026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e0a3d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc1026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46450/0 (socket says 172.21.15.79:46450) 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e0a3d700 1 -- 172.21.15.79:0/4182540295 learned_addr learned my addr 172.21.15.79:0/4182540295 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.084+0000 7fd4e0a3d700 1 -- 172.21.15.79:0/4182540295 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc102d40 con 0x7fd4dc1022b0 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4e0a3d700 1 --2- 172.21.15.79:0/4182540295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc1026c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00098e0 tx=0x7fd4d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef54b6d5880fe1d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4db7fe700 1 -- 172.21.15.79:0/4182540295 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d000b860 con 0x7fd4dc1022b0 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4db7fe700 1 -- 172.21.15.79:0/4182540295 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd4d000be30 con 0x7fd4dc1022b0 2024-01-19T01:14:39.885 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4db7fe700 1 -- 172.21.15.79:0/4182540295 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d0010380 con 0x7fd4dc1022b0 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/4182540295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 msgr2=0x7fd4dc1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4e2ca1700 1 --2- 172.21.15.79:0/4182540295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc1026c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00098e0 tx=0x7fd4d0009250 comp rx=0 tx=0).stop 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/4182540295 shutdown_connections 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.085+0000 7fd4e2ca1700 1 --2- 172.21.15.79:0/4182540295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc1026c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.086+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/4182540295 >> 172.21.15.79:0/4182540295 conn(0x7fd4dc0fda10 msgr2=0x7fd4dc0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.086+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/4182540295 shutdown_connections 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.086+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/4182540295 wait complete. 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.086+0000 7fd4e2ca1700 1 Processor -- start 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.086+0000 7fd4e2ca1700 1 -- start start 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.086+0000 7fd4e2ca1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.086+0000 7fd4e2ca1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc111690 con 0x7fd4dc1022b0 2024-01-19T01:14:39.886 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4e0a3d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4e0a3d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:46460/0 (socket says 172.21.15.79:46460) 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4e0a3d700 1 -- 172.21.15.79:0/2522784408 learned_addr learned my addr 172.21.15.79:0/2522784408 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4e0a3d700 1 -- 172.21.15.79:0/2522784408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00094f0 con 0x7fd4dc1022b0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4e0a3d700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc111150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0006b00 tx=0x7fd4d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d00109b0 con 0x7fd4dc1022b0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd4d0024440 con 0x7fd4dc1022b0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d001a5f0 con 0x7fd4dc1022b0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc111890 con 0x7fd4dc1022b0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.087+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc111ca0 con 0x7fd4dc1022b0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.088+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fd4d001a860 con 0x7fd4dc1022b0 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.088+0000 7fd4d97fa700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:39.887 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.088+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 --> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc039200 con 0x7fd4cc036640 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.088+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd4d004a5c0 con 0x7fd4dc1022b0 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.088+0000 7fd4dbfff700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 msgr2=0x7fd4cc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/430351410 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.088+0000 7fd4dbfff700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.289+0000 7fd4dbfff700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 msgr2=0x7fd4cc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/430351410 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.289+0000 7fd4dbfff700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.690+0000 7fd4dbfff700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 msgr2=0x7fd4cc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/430351410 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:34.690+0000 7fd4dbfff700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:35.491+0000 7fd4dbfff700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 msgr2=0x7fd4cc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/430351410 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:35.491+0000 7fd4dbfff700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:37.092+0000 7fd4dbfff700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 msgr2=0x7fd4cc038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/430351410 2024-01-19T01:14:39.888 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:37.092+0000 7fd4dbfff700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:38.244+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 6) v1 ==== 43048+0+0 (secure 0 0 0) 0x7fd4d000fdc0 con 0x7fd4dc1022b0 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:38.244+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 msgr2=0x7fd4cc038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:38.244+0000 7fd4d97fa700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.245+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7fd4d0024990 con 0x7fd4dc1022b0 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.245+0000 7fd4d97fa700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4cc039f60 0x7fd4cc03c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.245+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4cc039200 con 0x7fd4cc039f60 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.246+0000 7fd4dbfff700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4cc039f60 0x7fd4cc03c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.246+0000 7fd4dbfff700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4cc039f60 0x7fd4cc03c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80039d0 tx=0x7fd4c8009350 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.247+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fd4cc039200 con 0x7fd4cc039f60 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.253+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd4dc04bb80 con 0x7fd4cc039f60 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.253+0000 7fd4d97fa700 1 -- 172.21.15.79:0/2522784408 <== mgr.14116 v2:172.21.15.79:6800/3074418411 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd4dc04bb80 con 0x7fd4cc039f60 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4cc039f60 msgr2=0x7fd4cc03c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:39.889 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4cc039f60 0x7fd4cc03c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80039d0 tx=0x7fd4c8009350 comp rx=0 tx=0).stop 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 msgr2=0x7fd4dc111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc111150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0006b00 tx=0x7fd4d0009250 comp rx=0 tx=0).stop 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 shutdown_connections 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4cc039f60 0x7fd4cc03c340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd4dc1022b0 0x7fd4dc111150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 --2- 172.21.15.79:0/2522784408 >> [v2:172.21.15.79:6800/430351410,v1:172.21.15.79:6801/430351410] conn(0x7fd4cc036640 0x7fd4cc038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 >> 172.21.15.79:0/2522784408 conn(0x7fd4dc0fda10 msgr2=0x7fd4dc0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 shutdown_connections 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:39.254+0000 7fd4e2ca1700 1 -- 172.21.15.79:0/2522784408 wait complete. 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 5 is available 2024-01-19T01:14:39.890 INFO:teuthology.orchestra.run.smithi079.stdout:Setting orchestrator backend to cephadm... 2024-01-19T01:14:40.496 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:40 smithi079 ceph-mon[28374]: [19/Jan/2024:01:14:38] ENGINE Serving on https://172.21.15.79:7150 2024-01-19T01:14:40.496 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:40 smithi079 ceph-mon[28374]: [19/Jan/2024:01:14:38] ENGINE Bus STARTED 2024-01-19T01:14:40.497 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:40 smithi079 ceph-mon[28374]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-19T01:14:40.497 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:40 smithi079 ceph-mon[28374]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-19T01:14:41.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.285+0000 7fac69ca1700 1 Processor -- start 2024-01-19T01:14:41.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.285+0000 7fac69ca1700 1 -- start start 2024-01-19T01:14:41.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.285+0000 7fac69ca1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac64103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.285+0000 7fac69ca1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac64104310 con 0x7fac641039c0 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac637fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac64103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac637fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac64103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40478/0 (socket says 172.21.15.79:40478) 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac637fe700 1 -- 172.21.15.79:0/2278577178 learned_addr learned my addr 172.21.15.79:0/2278577178 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac637fe700 1 -- 172.21.15.79:0/2278577178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac64101fb0 con 0x7fac641039c0 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac637fe700 1 --2- 172.21.15.79:0/2278577178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac64103dd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fac580098e0 tx=0x7fac58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af18f4fa6295c0b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac627fc700 1 -- 172.21.15.79:0/2278577178 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac5800b860 con 0x7fac641039c0 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac627fc700 1 -- 172.21.15.79:0/2278577178 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fac5800be30 con 0x7fac641039c0 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.286+0000 7fac627fc700 1 -- 172.21.15.79:0/2278577178 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac58010380 con 0x7fac641039c0 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 -- 172.21.15.79:0/2278577178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 msgr2=0x7fac64103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 --2- 172.21.15.79:0/2278577178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac64103dd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fac580098e0 tx=0x7fac58009250 comp rx=0 tx=0).stop 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 -- 172.21.15.79:0/2278577178 shutdown_connections 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 --2- 172.21.15.79:0/2278577178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac64103dd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 -- 172.21.15.79:0/2278577178 >> 172.21.15.79:0/2278577178 conn(0x7fac640fda40 msgr2=0x7fac640ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:41.087 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 -- 172.21.15.79:0/2278577178 shutdown_connections 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 -- 172.21.15.79:0/2278577178 wait complete. 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.287+0000 7fac69ca1700 1 Processor -- start 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac69ca1700 1 -- start start 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac69ca1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac641110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac69ca1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac64111630 con 0x7fac641039c0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac637fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac641110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac637fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac641110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40480/0 (socket says 172.21.15.79:40480) 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac637fe700 1 -- 172.21.15.79:0/1621725654 learned_addr learned my addr 172.21.15.79:0/1621725654 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac637fe700 1 -- 172.21.15.79:0/1621725654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580094f0 con 0x7fac641039c0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.288+0000 7fac637fe700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac641110f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fac58006b00 tx=0x7fac58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.289+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac580109b0 con 0x7fac641039c0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.289+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac64111830 con 0x7fac641039c0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.289+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fac5801a430 con 0x7fac641039c0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.289+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac580245b0 con 0x7fac641039c0 2024-01-19T01:14:41.088 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.289+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac64111c40 con 0x7fac641039c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.289+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7fac58024830 con 0x7fac641039c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.290+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac64102d60 con 0x7fac641039c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.290+0000 7fac4bfff700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fac4c0364c0 0x7fac4c038970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.290+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fac5804ac40 con 0x7fac641039c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.290+0000 7fac62ffd700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fac4c0364c0 0x7fac4c038970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.290+0000 7fac62ffd700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fac4c0364c0 0x7fac4c038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac54009990 tx=0x7fac54006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.295+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fac58050020 con 0x7fac641039c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.416+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fac5801a9c0 con 0x7fac641039c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.495+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fac640ffdb0 con 0x7fac4c0364c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.502+0000 7fac4bfff700 1 -- 172.21.15.79:0/1621725654 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fac640ffdb0 con 0x7fac4c0364c0 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fac4c0364c0 msgr2=0x7fac4c038970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fac4c0364c0 0x7fac4c038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fac54009990 tx=0x7fac54006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 msgr2=0x7fac641110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac641110f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fac58006b00 tx=0x7fac58009250 comp rx=0 tx=0).stop 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 shutdown_connections 2024-01-19T01:14:41.089 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fac4c0364c0 0x7fac4c038970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:41.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 --2- 172.21.15.79:0/1621725654 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fac641039c0 0x7fac641110f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:41.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 >> 172.21.15.79:0/1621725654 conn(0x7fac640fda40 msgr2=0x7fac640ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:41.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 shutdown_connections 2024-01-19T01:14:41.090 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:40.504+0000 7fac69ca1700 1 -- 172.21.15.79:0/1621725654 wait complete. 2024-01-19T01:14:41.648 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:41 smithi079 ceph-mon[28374]: mgrmap e8: smithi079.miqczk(active, since 2s) 2024-01-19T01:14:41.648 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:41 smithi079 ceph-mon[28374]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:41.648 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:41 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:41.648 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:41 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout value unchanged 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.453+0000 7f4330ffa700 1 Processor -- start 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.453+0000 7f4330ffa700 1 -- start start 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f4330ffa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f4330ffa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c1031f0 con 0x7f432c1028a0 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f432a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f432a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40490/0 (socket says 172.21.15.79:40490) 2024-01-19T01:14:42.251 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f432a59c700 1 -- 172.21.15.79:0/1464132375 learned_addr learned my addr 172.21.15.79:0/1464132375 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f432a59c700 1 -- 172.21.15.79:0/1464132375 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c103330 con 0x7f432c1028a0 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f432a59c700 1 --2- 172.21.15.79:0/1464132375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c102cb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f431c0098e0 tx=0x7f431c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c31bffb7ba1ce4 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.454+0000 7f432959a700 1 -- 172.21.15.79:0/1464132375 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c00b860 con 0x7f432c1028a0 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f432959a700 1 -- 172.21.15.79:0/1464132375 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f431c00be30 con 0x7f432c1028a0 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f432959a700 1 -- 172.21.15.79:0/1464132375 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c010380 con 0x7f432c1028a0 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f4330ffa700 1 -- 172.21.15.79:0/1464132375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 msgr2=0x7f432c102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f4330ffa700 1 --2- 172.21.15.79:0/1464132375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c102cb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f431c0098e0 tx=0x7f431c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f4330ffa700 1 -- 172.21.15.79:0/1464132375 shutdown_connections 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f4330ffa700 1 --2- 172.21.15.79:0/1464132375 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c102cb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f4330ffa700 1 -- 172.21.15.79:0/1464132375 >> 172.21.15.79:0/1464132375 conn(0x7f432c0fc0b0 msgr2=0x7f432c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f4330ffa700 1 -- 172.21.15.79:0/1464132375 shutdown_connections 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.455+0000 7f4330ffa700 1 -- 172.21.15.79:0/1464132375 wait complete. 2024-01-19T01:14:42.252 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f4330ffa700 1 Processor -- start 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f4330ffa700 1 -- start start 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f4330ffa700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c113360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f4330ffa700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c1138a0 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f432a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c113360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f432a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c113360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40502/0 (socket says 172.21.15.79:40502) 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f432a59c700 1 -- 172.21.15.79:0/2512615150 learned_addr learned my addr 172.21.15.79:0/2512615150 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f432a59c700 1 -- 172.21.15.79:0/2512615150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c0094f0 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f432a59c700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c113360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f431c006b00 tx=0x7f431c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f431affd700 1 -- 172.21.15.79:0/2512615150 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c010a80 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f431affd700 1 -- 172.21.15.79:0/2512615150 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f431c01a430 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c113aa0 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.456+0000 7f431affd700 1 -- 172.21.15.79:0/2512615150 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f431c0245b0 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.457+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c113eb0 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.457+0000 7f431affd700 1 -- 172.21.15.79:0/2512615150 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f431c024830 con 0x7f432c1028a0 2024-01-19T01:14:42.253 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.457+0000 7f431affd700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f43140365f0 0x7f4314038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.457+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c04bb80 con 0x7f432c1028a0 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.457+0000 7f431affd700 1 -- 172.21.15.79:0/2512615150 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f431c04ab30 con 0x7f432c1028a0 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.457+0000 7f4329d9b700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f43140365f0 0x7f4314038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.458+0000 7f4329d9b700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f43140365f0 0x7f4314038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4320009990 tx=0x7f4320006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.462+0000 7f431affd700 1 -- 172.21.15.79:0/2512615150 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f431c018350 con 0x7f432c1028a0 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.647+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f432c10d1c0 con 0x7f43140365f0 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.647+0000 7f431affd700 1 -- 172.21.15.79:0/2512615150 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f432c10d1c0 con 0x7f43140365f0 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f43140365f0 msgr2=0x7f4314038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f43140365f0 0x7f4314038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4320009990 tx=0x7f4320006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 msgr2=0x7f432c113360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c113360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f431c006b00 tx=0x7f431c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 shutdown_connections 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f43140365f0 0x7f4314038aa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 --2- 172.21.15.79:0/2512615150 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f432c1028a0 0x7f432c113360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:42.254 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.649+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 >> 172.21.15.79:0/2512615150 conn(0x7f432c0fc0b0 msgr2=0x7f432c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:42.255 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.650+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 shutdown_connections 2024-01-19T01:14:42.255 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:41.650+0000 7f4330ffa700 1 -- 172.21.15.79:0/2512615150 wait complete. 2024-01-19T01:14:42.255 INFO:teuthology.orchestra.run.smithi079.stdout:Generating ssh key... 2024-01-19T01:14:42.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:42 smithi079 ceph-mon[28374]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:43.925 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e36a4700 1 Processor -- start 2024-01-19T01:14:43.925 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e36a4700 1 -- start start 2024-01-19T01:14:43.925 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e36a4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc107300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e36a4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc102cf0 con 0x7f32dc076c00 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e1440700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e1440700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc107300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40506/0 (socket says 172.21.15.79:40506) 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e1440700 1 -- 172.21.15.79:0/2481293827 learned_addr learned my addr 172.21.15.79:0/2481293827 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.693+0000 7f32e1440700 1 -- 172.21.15.79:0/2481293827 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc107840 con 0x7f32dc076c00 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32e1440700 1 --2- 172.21.15.79:0/2481293827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc107300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f32d80098e0 tx=0x7f32d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c877ce1241182552 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32d3fff700 1 -- 172.21.15.79:0/2481293827 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d800b860 con 0x7f32dc076c00 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32d3fff700 1 -- 172.21.15.79:0/2481293827 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f32d800be30 con 0x7f32dc076c00 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32d3fff700 1 -- 172.21.15.79:0/2481293827 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d8010380 con 0x7f32dc076c00 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32e36a4700 1 -- 172.21.15.79:0/2481293827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 msgr2=0x7f32dc107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32e36a4700 1 --2- 172.21.15.79:0/2481293827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc107300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f32d80098e0 tx=0x7f32d8009250 comp rx=0 tx=0).stop 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32e36a4700 1 -- 172.21.15.79:0/2481293827 shutdown_connections 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32e36a4700 1 --2- 172.21.15.79:0/2481293827 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc107300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.694+0000 7f32e36a4700 1 -- 172.21.15.79:0/2481293827 >> 172.21.15.79:0/2481293827 conn(0x7f32dc0fc950 msgr2=0x7f32dc0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:43.926 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e36a4700 1 -- 172.21.15.79:0/2481293827 shutdown_connections 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e36a4700 1 -- 172.21.15.79:0/2481293827 wait complete. 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e36a4700 1 Processor -- start 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e36a4700 1 -- start start 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e36a4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e36a4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32dc1138b0 con 0x7f32dc076c00 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e1440700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e1440700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc113370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40512/0 (socket says 172.21.15.79:40512) 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.695+0000 7f32e1440700 1 -- 172.21.15.79:0/690726922 learned_addr learned my addr 172.21.15.79:0/690726922 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.696+0000 7f32e1440700 1 -- 172.21.15.79:0/690726922 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80094f0 con 0x7f32dc076c00 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.696+0000 7f32e1440700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc113370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006d40 tx=0x7f32d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.696+0000 7f32d1ffb700 1 -- 172.21.15.79:0/690726922 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d80109b0 con 0x7f32dc076c00 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.696+0000 7f32d1ffb700 1 -- 172.21.15.79:0/690726922 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f32d8024440 con 0x7f32dc076c00 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.696+0000 7f32d1ffb700 1 -- 172.21.15.79:0/690726922 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d801a5f0 con 0x7f32dc076c00 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.696+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32dc113ab0 con 0x7f32dc076c00 2024-01-19T01:14:43.927 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.696+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32dc113ec0 con 0x7f32dc076c00 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.697+0000 7f32d1ffb700 1 -- 172.21.15.79:0/690726922 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f32d8024970 con 0x7f32dc076c00 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.697+0000 7f32d1ffb700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f32c8036640 0x7f32c8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.697+0000 7f32d1ffb700 1 -- 172.21.15.79:0/690726922 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f32d804a450 con 0x7f32dc076c00 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.697+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32dc04bb80 con 0x7f32dc076c00 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.697+0000 7f32e0c3f700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f32c8036640 0x7f32c8038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.697+0000 7f32e0c3f700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f32c8036640 0x7f32c8038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009990 tx=0x7f32cc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.702+0000 7f32d1ffb700 1 -- 172.21.15.79:0/690726922 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f32d8024790 con 0x7f32dc076c00 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:42.886+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f32dc10cf20 con 0x7f32c8036640 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.321+0000 7f32d1ffb700 1 -- 172.21.15.79:0/690726922 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f32dc10cf20 con 0x7f32c8036640 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f32c8036640 msgr2=0x7f32c8038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f32c8036640 0x7f32c8038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009990 tx=0x7f32cc006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 msgr2=0x7f32dc113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc113370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006d40 tx=0x7f32d8009250 comp rx=0 tx=0).stop 2024-01-19T01:14:43.928 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 shutdown_connections 2024-01-19T01:14:43.929 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f32c8036640 0x7f32c8038af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:43.929 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 --2- 172.21.15.79:0/690726922 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f32dc076c00 0x7f32dc113370 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:43.929 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 >> 172.21.15.79:0/690726922 conn(0x7f32dc0fc950 msgr2=0x7f32dc0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:43.929 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 shutdown_connections 2024-01-19T01:14:43.929 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:43.323+0000 7f32e36a4700 1 -- 172.21.15.79:0/690726922 wait complete. 2024-01-19T01:14:44.482 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:44 smithi079 ceph-mon[28374]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:44.482 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:44 smithi079 ceph-mon[28374]: Generating ssh key... 2024-01-19T01:14:44.482 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:44 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:44.482 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:44 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:45.081 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg0aK8WjoCR33NldTWJZDgUoAQgWnfu1uvosYtB1Oy79TNpmCYtIZV53iGu62d0/yZvyxcWAOhtccO6GlOpCk/WHIiDFk6vrST22fP7IqrDBxLHQDvftcggAME3MChuWpK6MKO4+YolIIcXFqnBWBcGgll7lxvkG3leSo+pJKCWePAbpaseAB4skdlBS3z/zl0UDxhZizogTxj1FiEsH+/iKweITTGvq+FZLYzmr6cca2c3fZKLGLCozEANWYyl0hXxgJW5Oykf/HECCt4SV5w8OTgCsaEEyymr+BnU0uGyjcoDj/UQnHKkakV2zEM7jeXmks4vC1/TZYhara690TTuVaEl6+H91bcV7pVJ+nEPfXuUl0OJ+9utOEwKUJtgU+pm4CUItkFsgUEqbtQtoFYyJFCeOvA5qf//T5iEIh8BNNjDROxLRi9BP3frVLla3/IlP2fUUgcWOCnaPE4bCQT/JOmQN0hGMoNA2zns1p4efJUgg3y/OLaWndyfa1WFP0= ceph-f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:14:45.081 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd4216af700 1 Processor -- start 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd4216af700 1 -- start start 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd4216af700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd4216af700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c104310 con 0x7fd41c1039c0 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd41affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd41affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40516/0 (socket says 172.21.15.79:40516) 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd41affd700 1 -- 172.21.15.79:0/2089004275 learned_addr learned my addr 172.21.15.79:0/2089004275 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.285+0000 7fd41affd700 1 -- 172.21.15.79:0/2089004275 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c101fb0 con 0x7fd41c1039c0 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd41affd700 1 --2- 172.21.15.79:0/2089004275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0098e0 tx=0x7fd40c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5837de80175128 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd419ffb700 1 -- 172.21.15.79:0/2089004275 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd40c00b860 con 0x7fd41c1039c0 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd419ffb700 1 -- 172.21.15.79:0/2089004275 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd40c00be30 con 0x7fd41c1039c0 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd419ffb700 1 -- 172.21.15.79:0/2089004275 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd40c010380 con 0x7fd41c1039c0 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd4216af700 1 -- 172.21.15.79:0/2089004275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 msgr2=0x7fd41c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:45.082 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd4216af700 1 --2- 172.21.15.79:0/2089004275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0098e0 tx=0x7fd40c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd4216af700 1 -- 172.21.15.79:0/2089004275 shutdown_connections 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd4216af700 1 --2- 172.21.15.79:0/2089004275 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c103dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd4216af700 1 -- 172.21.15.79:0/2089004275 >> 172.21.15.79:0/2089004275 conn(0x7fd41c0fda40 msgr2=0x7fd41c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd4216af700 1 -- 172.21.15.79:0/2089004275 shutdown_connections 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.286+0000 7fd4216af700 1 -- 172.21.15.79:0/2089004275 wait complete. 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.287+0000 7fd4216af700 1 Processor -- start 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.287+0000 7fd4216af700 1 -- start start 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.287+0000 7fd4216af700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c1110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.287+0000 7fd4216af700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c111630 con 0x7fd41c1039c0 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.287+0000 7fd41affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c1110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.287+0000 7fd41affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c1110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40526/0 (socket says 172.21.15.79:40526) 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.287+0000 7fd41affd700 1 -- 172.21.15.79:0/3794530921 learned_addr learned my addr 172.21.15.79:0/3794530921 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.288+0000 7fd41affd700 1 -- 172.21.15.79:0/3794530921 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c0094f0 con 0x7fd41c1039c0 2024-01-19T01:14:45.083 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.288+0000 7fd41affd700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c1110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd40c000c00 tx=0x7fd40c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.288+0000 7fd4037fe700 1 -- 172.21.15.79:0/3794530921 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd40c010a80 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.288+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41c111830 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.288+0000 7fd4037fe700 1 -- 172.21.15.79:0/3794530921 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd40c01a430 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.288+0000 7fd4037fe700 1 -- 172.21.15.79:0/3794530921 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd40c0245b0 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.288+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41c111c40 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.289+0000 7fd4037fe700 1 -- 172.21.15.79:0/3794530921 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fd40c024870 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.289+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41c102d60 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.289+0000 7fd4037fe700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4040365f0 0x7fd404038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.289+0000 7fd4037fe700 1 -- 172.21.15.79:0/3794530921 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd40c04a7c0 con 0x7fd41c1039c0 2024-01-19T01:14:45.084 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.289+0000 7fd41a7fc700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4040365f0 0x7fd404038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.290+0000 7fd41a7fc700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4040365f0 0x7fd404038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd410009990 tx=0x7fd410006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.294+0000 7fd4037fe700 1 -- 172.21.15.79:0/3794530921 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd40c018530 con 0x7fd41c1039c0 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.480+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd41c10adf0 con 0x7fd4040365f0 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.481+0000 7fd4037fe700 1 -- 172.21.15.79:0/3794530921 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd41c10adf0 con 0x7fd4040365f0 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4040365f0 msgr2=0x7fd404038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4040365f0 0x7fd404038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd410009990 tx=0x7fd410006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 msgr2=0x7fd41c1110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c1110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd40c000c00 tx=0x7fd40c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 shutdown_connections 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fd4040365f0 0x7fd404038aa0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 --2- 172.21.15.79:0/3794530921 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd41c1039c0 0x7fd41c1110f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.483+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 >> 172.21.15.79:0/3794530921 conn(0x7fd41c0fda40 msgr2=0x7fd41c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:45.085 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.484+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 shutdown_connections 2024-01-19T01:14:45.086 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:44.484+0000 7fd4216af700 1 -- 172.21.15.79:0/3794530921 wait complete. 2024-01-19T01:14:45.086 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-19T01:14:45.086 INFO:teuthology.orchestra.run.smithi079.stdout:Adding key to root@localhost authorized_keys... 2024-01-19T01:14:45.086 INFO:teuthology.orchestra.run.smithi079.stdout:Adding host smithi079... 2024-01-19T01:14:45.686 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:45 smithi079 ceph-mon[28374]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:46.676 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:46 smithi079 ceph-mon[28374]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:47.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:47 smithi079 ceph-mon[28374]: Deploying cephadm binary to smithi079 2024-01-19T01:14:48.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Added host 'smithi079' with addr '172.21.15.79' 2024-01-19T01:14:48.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.488+0000 7f303c323700 1 Processor -- start 2024-01-19T01:14:48.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.488+0000 7f303c323700 1 -- start start 2024-01-19T01:14:48.280 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.489+0000 7f303c323700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f30341015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.489+0000 7f303c323700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3034101b20 con 0x7f30341011d0 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.489+0000 7f303a0bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f30341015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.489+0000 7f303a0bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f30341015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40534/0 (socket says 172.21.15.79:40534) 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.489+0000 7f303a0bf700 1 -- 172.21.15.79:0/2224154584 learned_addr learned my addr 172.21.15.79:0/2224154584 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.489+0000 7f303a0bf700 1 -- 172.21.15.79:0/2224154584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3034101c60 con 0x7f30341011d0 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.489+0000 7f303a0bf700 1 --2- 172.21.15.79:0/2224154584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f30341015e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30300098e0 tx=0x7f3030009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f539051511cbaf server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f30390bd700 1 -- 172.21.15.79:0/2224154584 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f303000b860 con 0x7f30341011d0 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f30390bd700 1 -- 172.21.15.79:0/2224154584 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f303000be30 con 0x7f30341011d0 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f30390bd700 1 -- 172.21.15.79:0/2224154584 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3030010380 con 0x7f30341011d0 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f303c323700 1 -- 172.21.15.79:0/2224154584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 msgr2=0x7f30341015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f303c323700 1 --2- 172.21.15.79:0/2224154584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f30341015e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f30300098e0 tx=0x7f3030009250 comp rx=0 tx=0).stop 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f303c323700 1 -- 172.21.15.79:0/2224154584 shutdown_connections 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f303c323700 1 --2- 172.21.15.79:0/2224154584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f30341015e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f303c323700 1 -- 172.21.15.79:0/2224154584 >> 172.21.15.79:0/2224154584 conn(0x7f30340fc950 msgr2=0x7f30340fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:48.281 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f303c323700 1 -- 172.21.15.79:0/2224154584 shutdown_connections 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.490+0000 7f303c323700 1 -- 172.21.15.79:0/2224154584 wait complete. 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303c323700 1 Processor -- start 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303c323700 1 -- start start 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303c323700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f3034111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303c323700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3034111680 con 0x7f30341011d0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303a0bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f3034111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303a0bf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f3034111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:40548/0 (socket says 172.21.15.79:40548) 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303a0bf700 1 -- 172.21.15.79:0/3791689127 learned_addr learned my addr 172.21.15.79:0/3791689127 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303a0bf700 1 -- 172.21.15.79:0/3791689127 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300094f0 con 0x7f30341011d0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.491+0000 7f303a0bf700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f3034111140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3030006b00 tx=0x7f3030009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.492+0000 7f302affd700 1 -- 172.21.15.79:0/3791689127 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30300109b0 con 0x7f30341011d0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.492+0000 7f302affd700 1 -- 172.21.15.79:0/3791689127 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3030024440 con 0x7f30341011d0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.492+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3034111880 con 0x7f30341011d0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.492+0000 7f302affd700 1 -- 172.21.15.79:0/3791689127 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f303001a5f0 con 0x7f30341011d0 2024-01-19T01:14:48.282 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.492+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3034111c90 con 0x7f30341011d0 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.493+0000 7f302affd700 1 -- 172.21.15.79:0/3791689127 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f3030024990 con 0x7f30341011d0 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.493+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303404bb80 con 0x7f30341011d0 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.493+0000 7f302affd700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f3020036640 0x7f3020038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.493+0000 7f302affd700 1 -- 172.21.15.79:0/3791689127 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f303004a570 con 0x7f30341011d0 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.493+0000 7f30398be700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f3020036640 0x7f3020038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.493+0000 7f30398be700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f3020036640 0x7f3020038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3024009990 tx=0x7f3024006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.498+0000 7f302affd700 1 -- 172.21.15.79:0/3791689127 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3030006d70 con 0x7f30341011d0 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:45.685+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}) v1 -- 0x7f30340fec50 con 0x7f3020036640 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.718+0000 7f302affd700 1 -- 172.21.15.79:0/3791689127 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f30340fec50 con 0x7f3020036640 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f3020036640 msgr2=0x7f3020038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f3020036640 0x7f3020038af0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3024009990 tx=0x7f3024006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 msgr2=0x7f3034111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f3034111140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3030006b00 tx=0x7f3030009250 comp rx=0 tx=0).stop 2024-01-19T01:14:48.283 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 shutdown_connections 2024-01-19T01:14:48.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f3020036640 0x7f3020038af0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:48.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 --2- 172.21.15.79:0/3791689127 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f30341011d0 0x7f3034111140 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:48.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.720+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 >> 172.21.15.79:0/3791689127 conn(0x7f30340fc950 msgr2=0x7f3034076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:48.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.721+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 shutdown_connections 2024-01-19T01:14:48.284 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:47.721+0000 7f303c323700 1 -- 172.21.15.79:0/3791689127 wait complete. 2024-01-19T01:14:48.284 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying mon service with default placement... 2024-01-19T01:14:48.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:48 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:48 smithi079 ceph-mon[28374]: Added host smithi079 2024-01-19T01:14:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:48 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.648+0000 7f5e11f29700 1 Processor -- start 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.648+0000 7f5e11f29700 1 -- start start 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.648+0000 7f5e11f29700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.649+0000 7f5e11f29700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c104240 con 0x7f5e0c1038f0 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.649+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.649+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58634/0 (socket says 172.21.15.79:58634) 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.649+0000 7f5e0b7fe700 1 -- 172.21.15.79:0/1471159618 learned_addr learned my addr 172.21.15.79:0/1471159618 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.649+0000 7f5e0b7fe700 1 -- 172.21.15.79:0/1471159618 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c104380 con 0x7f5e0c1038f0 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.649+0000 7f5e0b7fe700 1 --2- 172.21.15.79:0/1471159618 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c103d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098e0 tx=0x7f5dfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447e975514098a6d server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e0affd700 1 -- 172.21.15.79:0/1471159618 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dfc00b860 con 0x7f5e0c1038f0 2024-01-19T01:14:49.440 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e0affd700 1 -- 172.21.15.79:0/1471159618 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5dfc00be30 con 0x7f5e0c1038f0 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e0affd700 1 -- 172.21.15.79:0/1471159618 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dfc010380 con 0x7f5e0c1038f0 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e11f29700 1 -- 172.21.15.79:0/1471159618 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 msgr2=0x7f5e0c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e11f29700 1 --2- 172.21.15.79:0/1471159618 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c103d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098e0 tx=0x7f5dfc009250 comp rx=0 tx=0).stop 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e11f29700 1 -- 172.21.15.79:0/1471159618 shutdown_connections 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e11f29700 1 --2- 172.21.15.79:0/1471159618 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c103d00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e11f29700 1 -- 172.21.15.79:0/1471159618 >> 172.21.15.79:0/1471159618 conn(0x7f5e0c0fc0b0 msgr2=0x7f5e0c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e11f29700 1 -- 172.21.15.79:0/1471159618 shutdown_connections 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.650+0000 7f5e11f29700 1 -- 172.21.15.79:0/1471159618 wait complete. 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e11f29700 1 Processor -- start 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e11f29700 1 -- start start 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e11f29700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c111100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e11f29700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c111640 con 0x7f5e0c1038f0 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c111100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e0b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c111100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58648/0 (socket says 172.21.15.79:58648) 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e0b7fe700 1 -- 172.21.15.79:0/3394607928 learned_addr learned my addr 172.21.15.79:0/3394607928 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:49.441 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.651+0000 7f5e0b7fe700 1 -- 172.21.15.79:0/3394607928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc0094f0 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.652+0000 7f5e0b7fe700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c111100 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc006b00 tx=0x7f5dfc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.652+0000 7f5e08ff9700 1 -- 172.21.15.79:0/3394607928 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dfc0109b0 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.652+0000 7f5e08ff9700 1 -- 172.21.15.79:0/3394607928 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5dfc024440 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.652+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c111840 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.652+0000 7f5e08ff9700 1 -- 172.21.15.79:0/3394607928 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5dfc01a5f0 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.652+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c111c50 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.653+0000 7f5e08ff9700 1 -- 172.21.15.79:0/3394607928 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5dfc01a860 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.653+0000 7f5e08ff9700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5df8036640 0x7f5df8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.653+0000 7f5e08ff9700 1 -- 172.21.15.79:0/3394607928 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5dfc04a5a0 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.653+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c04bb80 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.653+0000 7f5e02ffd700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5df8036640 0x7f5df8038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.653+0000 7f5e02ffd700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5df8036640 0x7f5df8038af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009990 tx=0x7f5df4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.658+0000 7f5e08ff9700 1 -- 172.21.15.79:0/3394607928 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5dfc01ab60 con 0x7f5e0c1038f0 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.856+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e0c0008d0 con 0x7f5df8036640 2024-01-19T01:14:49.442 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.863+0000 7f5e08ff9700 1 -- 172.21.15.79:0/3394607928 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5e0c0008d0 con 0x7f5df8036640 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5df8036640 msgr2=0x7f5df8038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5df8036640 0x7f5df8038af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009990 tx=0x7f5df4006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 msgr2=0x7f5e0c111100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c111100 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc006b00 tx=0x7f5dfc009250 comp rx=0 tx=0).stop 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 shutdown_connections 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5df8036640 0x7f5df8038af0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 --2- 172.21.15.79:0/3394607928 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5e0c1038f0 0x7f5e0c111100 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 >> 172.21.15.79:0/3394607928 conn(0x7f5e0c0fc0b0 msgr2=0x7f5e0c0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 shutdown_connections 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:48.865+0000 7f5e11f29700 1 -- 172.21.15.79:0/3394607928 wait complete. 2024-01-19T01:14:49.443 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying mgr service with default placement... 2024-01-19T01:14:49.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:49 smithi079 ceph-mon[28374]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:49.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:49 smithi079 ceph-mon[28374]: Saving service mon spec with placement count:5 2024-01-19T01:14:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:49 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:51.235 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:51 smithi079 ceph-mon[28374]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:51.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:51 smithi079 ceph-mon[28374]: Saving service mgr spec with placement count:2 2024-01-19T01:14:51.236 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:51 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:51.502 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-01-19T01:14:51.502 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.836+0000 7f74a5392700 1 Processor -- start 2024-01-19T01:14:51.502 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.836+0000 7f74a5392700 1 -- start start 2024-01-19T01:14:51.502 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.836+0000 7f74a5392700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a01026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.836+0000 7f74a5392700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a0102c20 con 0x7f74a01022d0 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.836+0000 7f749effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a01026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.836+0000 7f749effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a01026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58656/0 (socket says 172.21.15.79:58656) 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.836+0000 7f749effd700 1 -- 172.21.15.79:0/2982200222 learned_addr learned my addr 172.21.15.79:0/2982200222 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.837+0000 7f749effd700 1 -- 172.21.15.79:0/2982200222 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0102d60 con 0x7f74a01022d0 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.837+0000 7f749effd700 1 --2- 172.21.15.79:0/2982200222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a01026e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74900098e0 tx=0x7f7490009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8ba612ac5b8177 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.837+0000 7f749dffb700 1 -- 172.21.15.79:0/2982200222 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f749000b860 con 0x7f74a01022d0 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.837+0000 7f749dffb700 1 -- 172.21.15.79:0/2982200222 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f749000be30 con 0x7f74a01022d0 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.837+0000 7f749dffb700 1 -- 172.21.15.79:0/2982200222 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7490010380 con 0x7f74a01022d0 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.837+0000 7f74a5392700 1 -- 172.21.15.79:0/2982200222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 msgr2=0x7f74a01026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.837+0000 7f74a5392700 1 --2- 172.21.15.79:0/2982200222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a01026e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74900098e0 tx=0x7f7490009250 comp rx=0 tx=0).stop 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 -- 172.21.15.79:0/2982200222 shutdown_connections 2024-01-19T01:14:51.503 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 --2- 172.21.15.79:0/2982200222 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a01026e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 -- 172.21.15.79:0/2982200222 >> 172.21.15.79:0/2982200222 conn(0x7f74a00fda50 msgr2=0x7f74a00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 -- 172.21.15.79:0/2982200222 shutdown_connections 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 -- 172.21.15.79:0/2982200222 wait complete. 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 Processor -- start 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 -- start start 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a0111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f74a5392700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a0111680 con 0x7f74a01022d0 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f749effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a0111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f749effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a0111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58664/0 (socket says 172.21.15.79:58664) 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.838+0000 7f749effd700 1 -- 172.21.15.79:0/344933034 learned_addr learned my addr 172.21.15.79:0/344933034 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.839+0000 7f749effd700 1 -- 172.21.15.79:0/344933034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900094f0 con 0x7f74a01022d0 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.839+0000 7f749effd700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a0111140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7490000c00 tx=0x7f7490009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.839+0000 7f74877fe700 1 -- 172.21.15.79:0/344933034 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7490010a80 con 0x7f74a01022d0 2024-01-19T01:14:51.504 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.839+0000 7f74877fe700 1 -- 172.21.15.79:0/344933034 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f749001a430 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.839+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a0111880 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.839+0000 7f74877fe700 1 -- 172.21.15.79:0/344933034 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74900245b0 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.839+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a0111c90 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.840+0000 7f74877fe700 1 -- 172.21.15.79:0/344933034 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f74900247f0 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.840+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a004bb80 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.840+0000 7f74877fe700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f74880365f0 0x7f7488038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.840+0000 7f74877fe700 1 -- 172.21.15.79:0/344933034 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f749004acf0 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.840+0000 7f749e7fc700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f74880365f0 0x7f7488038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.841+0000 7f749e7fc700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f74880365f0 0x7f7488038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7494009990 tx=0x7f7494006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:49.845+0000 7f74877fe700 1 -- 172.21.15.79:0/344933034 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f749004f070 con 0x7f74a01022d0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.044+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f74a010afc0 con 0x7f74880365f0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.051+0000 7f74877fe700 1 -- 172.21.15.79:0/344933034 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f74a010afc0 con 0x7f74880365f0 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f74880365f0 msgr2=0x7f7488038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:51.505 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f74880365f0 0x7f7488038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7494009990 tx=0x7f7494006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 msgr2=0x7f74a0111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a0111140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7490000c00 tx=0x7f7490009250 comp rx=0 tx=0).stop 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 shutdown_connections 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f74880365f0 0x7f7488038aa0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 --2- 172.21.15.79:0/344933034 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f74a01022d0 0x7f74a0111140 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 >> 172.21.15.79:0/344933034 conn(0x7f74a00fda50 msgr2=0x7f74a00ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 shutdown_connections 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:50.053+0000 7f74a5392700 1 -- 172.21.15.79:0/344933034 wait complete. 2024-01-19T01:14:51.506 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying crash service with default placement... 2024-01-19T01:14:52.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:52 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:52.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:52 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:53.129 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.928+0000 7f07d9895700 1 Processor -- start 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.928+0000 7f07d9895700 1 -- start start 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.928+0000 7f07d9895700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d41015d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.928+0000 7f07d9895700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4101b10 con 0x7f07d41011c0 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.928+0000 7f07d2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d41015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.928+0000 7f07d2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d41015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58666/0 (socket says 172.21.15.79:58666) 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.928+0000 7f07d2ffd700 1 -- 172.21.15.79:0/1253298942 learned_addr learned my addr 172.21.15.79:0/1253298942 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.929+0000 7f07d2ffd700 1 -- 172.21.15.79:0/1253298942 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d4101c50 con 0x7f07d41011c0 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.929+0000 7f07d2ffd700 1 --2- 172.21.15.79:0/1253298942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d41015d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07c40098e0 tx=0x7f07c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd1fa3ac5c14798 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.929+0000 7f07d1ffb700 1 -- 172.21.15.79:0/1253298942 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c400b860 con 0x7f07d41011c0 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.929+0000 7f07d1ffb700 1 -- 172.21.15.79:0/1253298942 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07c400be70 con 0x7f07d41011c0 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.929+0000 7f07d1ffb700 1 -- 172.21.15.79:0/1253298942 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4010380 con 0x7f07d41011c0 2024-01-19T01:14:53.130 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 -- 172.21.15.79:0/1253298942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 msgr2=0x7f07d41015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 --2- 172.21.15.79:0/1253298942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d41015d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f07c40098e0 tx=0x7f07c4009250 comp rx=0 tx=0).stop 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 -- 172.21.15.79:0/1253298942 shutdown_connections 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 --2- 172.21.15.79:0/1253298942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d41015d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 -- 172.21.15.79:0/1253298942 >> 172.21.15.79:0/1253298942 conn(0x7f07d40fc980 msgr2=0x7f07d40fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 -- 172.21.15.79:0/1253298942 shutdown_connections 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 -- 172.21.15.79:0/1253298942 wait complete. 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 Processor -- start 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.930+0000 7f07d9895700 1 -- start start 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07d9895700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d4116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07d9895700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4111c30 con 0x7f07d41011c0 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07d2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d4116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07d2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d4116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58670/0 (socket says 172.21.15.79:58670) 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07d2ffd700 1 -- 172.21.15.79:0/2311028199 learned_addr learned my addr 172.21.15.79:0/2311028199 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07d2ffd700 1 -- 172.21.15.79:0/2311028199 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c40094f0 con 0x7f07d41011c0 2024-01-19T01:14:53.131 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07d2ffd700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d4116240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07c4011b60 tx=0x7f07c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.931+0000 7f07bb7fe700 1 -- 172.21.15.79:0/2311028199 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c4010850 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.932+0000 7f07bb7fe700 1 -- 172.21.15.79:0/2311028199 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07c4010e60 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.932+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d4076c50 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.932+0000 7f07bb7fe700 1 -- 172.21.15.79:0/2311028199 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c401ae20 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.932+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d4116780 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.932+0000 7f07bb7fe700 1 -- 172.21.15.79:0/2311028199 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f07c402d3f0 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.932+0000 7f07bb7fe700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f07bc0365f0 0x7f07bc038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.933+0000 7f07bb7fe700 1 -- 172.21.15.79:0/2311028199 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f07c404a200 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.933+0000 7f07d27fc700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f07bc0365f0 0x7f07bc038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.933+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d410a080 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.933+0000 7f07d27fc700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f07bc0365f0 0x7f07bc038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009990 tx=0x7f07c8006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:51.937+0000 7f07bb7fe700 1 -- 172.21.15.79:0/2311028199 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f07c402b8e0 con 0x7f07d41011c0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.180+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f07d402b150 con 0x7f07bc0365f0 2024-01-19T01:14:53.132 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.185+0000 7f07bb7fe700 1 -- 172.21.15.79:0/2311028199 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f07d402b150 con 0x7f07bc0365f0 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.187+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f07bc0365f0 msgr2=0x7f07bc038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.187+0000 7f07d9895700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f07bc0365f0 0x7f07bc038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009990 tx=0x7f07c8006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.187+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 msgr2=0x7f07d4116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.187+0000 7f07d9895700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d4116240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f07c4011b60 tx=0x7f07c4009250 comp rx=0 tx=0).stop 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.187+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 shutdown_connections 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.188+0000 7f07d9895700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f07bc0365f0 0x7f07bc038aa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.188+0000 7f07d9895700 1 --2- 172.21.15.79:0/2311028199 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f07d41011c0 0x7f07d4116240 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.188+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 >> 172.21.15.79:0/2311028199 conn(0x7f07d40fc980 msgr2=0x7f07d40fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.188+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 shutdown_connections 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:52.188+0000 7f07d9895700 1 -- 172.21.15.79:0/2311028199 wait complete. 2024-01-19T01:14:53.133 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying prometheus service with default placement... 2024-01-19T01:14:53.441 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:53 smithi079 ceph-mon[28374]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:53.441 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:53 smithi079 ceph-mon[28374]: Saving service crash spec with placement * 2024-01-19T01:14:53.441 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:53 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:54.311 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-01-19T01:14:54.311 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.541+0000 7fb2a0b85700 1 Processor -- start 2024-01-19T01:14:54.311 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.541+0000 7fb2a0b85700 1 -- start start 2024-01-19T01:14:54.311 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb2a0b85700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:54.311 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb2a0b85700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c105420 con 0x7fb29c070fc0 2024-01-19T01:14:54.311 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58674/0 (socket says 172.21.15.79:58674) 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29a59c700 1 -- 172.21.15.79:0/184295585 learned_addr learned my addr 172.21.15.79:0/184295585 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29a59c700 1 -- 172.21.15.79:0/184295585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c105560 con 0x7fb29c070fc0 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29a59c700 1 --2- 172.21.15.79:0/184295585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c104ee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0098e0 tx=0x7fb28c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40fa11f2bfac98f server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29959a700 1 -- 172.21.15.79:0/184295585 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28c00b860 con 0x7fb29c070fc0 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29959a700 1 -- 172.21.15.79:0/184295585 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb28c00be70 con 0x7fb29c070fc0 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.542+0000 7fb29959a700 1 -- 172.21.15.79:0/184295585 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28c010380 con 0x7fb29c070fc0 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 -- 172.21.15.79:0/184295585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 msgr2=0x7fb29c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 --2- 172.21.15.79:0/184295585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c104ee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0098e0 tx=0x7fb28c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 -- 172.21.15.79:0/184295585 shutdown_connections 2024-01-19T01:14:54.312 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 --2- 172.21.15.79:0/184295585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c104ee0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 -- 172.21.15.79:0/184295585 >> 172.21.15.79:0/184295585 conn(0x7fb29c0fc970 msgr2=0x7fb29c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 -- 172.21.15.79:0/184295585 shutdown_connections 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 -- 172.21.15.79:0/184295585 wait complete. 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.543+0000 7fb2a0b85700 1 Processor -- start 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb2a0b85700 1 -- start start 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb2a0b85700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c1111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb2a0b85700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c111700 con 0x7fb29c070fc0 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb29a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c1111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb29a59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c1111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58690/0 (socket says 172.21.15.79:58690) 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb29a59c700 1 -- 172.21.15.79:0/129231164 learned_addr learned my addr 172.21.15.79:0/129231164 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb29a59c700 1 -- 172.21.15.79:0/129231164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c0094f0 con 0x7fb29c070fc0 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb29a59c700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c1111c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006d40 tx=0x7fb28c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb28affd700 1 -- 172.21.15.79:0/129231164 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28c010960 con 0x7fb29c070fc0 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c111900 con 0x7fb29c070fc0 2024-01-19T01:14:54.313 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb28affd700 1 -- 172.21.15.79:0/129231164 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb28c01a430 con 0x7fb29c070fc0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb28affd700 1 -- 172.21.15.79:0/129231164 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb28c0245b0 con 0x7fb29c070fc0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.544+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c111d10 con 0x7fb29c070fc0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.545+0000 7fb28affd700 1 -- 172.21.15.79:0/129231164 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb28c02d3f0 con 0x7fb29c070fc0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.545+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c04bb80 con 0x7fb29c070fc0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.545+0000 7fb28affd700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb2840365f0 0x7fb284038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.545+0000 7fb28affd700 1 -- 172.21.15.79:0/129231164 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb28c04a280 con 0x7fb29c070fc0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.546+0000 7fb299d9b700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb2840365f0 0x7fb284038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.546+0000 7fb299d9b700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb2840365f0 0x7fb284038aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb290009990 tx=0x7fb290006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.550+0000 7fb28affd700 1 -- 172.21.15.79:0/129231164 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb28c018b80 con 0x7fb29c070fc0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.759+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb29c0008d0 con 0x7fb2840365f0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.764+0000 7fb28affd700 1 -- 172.21.15.79:0/129231164 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fb29c0008d0 con 0x7fb2840365f0 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb2840365f0 msgr2=0x7fb284038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb2840365f0 0x7fb284038aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb290009990 tx=0x7fb290006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:54.314 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 msgr2=0x7fb29c1111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c1111c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006d40 tx=0x7fb28c009250 comp rx=0 tx=0).stop 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 shutdown_connections 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb2840365f0 0x7fb284038aa0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 --2- 172.21.15.79:0/129231164 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb29c070fc0 0x7fb29c1111c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 >> 172.21.15.79:0/129231164 conn(0x7fb29c0fc970 msgr2=0x7fb29c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 shutdown_connections 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:53.766+0000 7fb2a0b85700 1 -- 172.21.15.79:0/129231164 wait complete. 2024-01-19T01:14:54.315 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying grafana service with default placement... 2024-01-19T01:14:54.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:54 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:54.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:54 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:54.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:54 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:55 smithi079 ceph-mon[28374]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:55 smithi079 ceph-mon[28374]: Saving service prometheus spec with placement count:1 2024-01-19T01:14:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:55 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:56.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-01-19T01:14:56.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.780+0000 7f1383627700 1 Processor -- start 2024-01-19T01:14:56.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.780+0000 7f1383627700 1 -- start start 2024-01-19T01:14:56.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.780+0000 7f1383627700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:56.550 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.780+0000 7f1383627700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c0790f0 con 0x7f137c078ce0 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.780+0000 7f13813c3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.780+0000 7f13813c3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c0771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58696/0 (socket says 172.21.15.79:58696) 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.780+0000 7f13813c3700 1 -- 172.21.15.79:0/199270517 learned_addr learned my addr 172.21.15.79:0/199270517 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.781+0000 7f13813c3700 1 -- 172.21.15.79:0/199270517 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c077730 con 0x7f137c078ce0 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.781+0000 7f13813c3700 1 --2- 172.21.15.79:0/199270517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c0771f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13780098e0 tx=0x7f1378009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d972404851f68dd server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.781+0000 7f1373fff700 1 -- 172.21.15.79:0/199270517 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f137800b860 con 0x7f137c078ce0 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.781+0000 7f1373fff700 1 -- 172.21.15.79:0/199270517 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f137800be70 con 0x7f137c078ce0 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.781+0000 7f1373fff700 1 -- 172.21.15.79:0/199270517 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378010380 con 0x7f137c078ce0 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.781+0000 7f1383627700 1 -- 172.21.15.79:0/199270517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 msgr2=0x7f137c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.781+0000 7f1383627700 1 --2- 172.21.15.79:0/199270517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c0771f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13780098e0 tx=0x7f1378009250 comp rx=0 tx=0).stop 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.782+0000 7f1383627700 1 -- 172.21.15.79:0/199270517 shutdown_connections 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.782+0000 7f1383627700 1 --2- 172.21.15.79:0/199270517 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c0771f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:56.551 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.782+0000 7f1383627700 1 -- 172.21.15.79:0/199270517 >> 172.21.15.79:0/199270517 conn(0x7f137c0fda10 msgr2=0x7f137c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.782+0000 7f1383627700 1 -- 172.21.15.79:0/199270517 shutdown_connections 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.782+0000 7f1383627700 1 -- 172.21.15.79:0/199270517 wait complete. 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.782+0000 7f1383627700 1 Processor -- start 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.782+0000 7f1383627700 1 -- start start 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f1383627700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f1383627700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c113870 con 0x7f137c078ce0 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f13813c3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f13813c3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58698/0 (socket says 172.21.15.79:58698) 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f13813c3700 1 -- 172.21.15.79:0/3630418187 learned_addr learned my addr 172.21.15.79:0/3630418187 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f13813c3700 1 -- 172.21.15.79:0/3630418187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13780094f0 con 0x7f137c078ce0 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f13813c3700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c113330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13780094c0 tx=0x7f1378009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:56.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f1371ffb700 1 -- 172.21.15.79:0/3630418187 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378010850 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f1371ffb700 1 -- 172.21.15.79:0/3630418187 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1378010e60 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c113a70 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.783+0000 7f1371ffb700 1 -- 172.21.15.79:0/3630418187 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13780243f0 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.784+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c113e80 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.785+0000 7f1371ffb700 1 -- 172.21.15.79:0/3630418187 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f137801a430 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.785+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c04bb80 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.785+0000 7f1371ffb700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f1368036640 0x7f1368038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.785+0000 7f1371ffb700 1 -- 172.21.15.79:0/3630418187 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f137804a5f0 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.785+0000 7f1380bc2700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f1368036640 0x7f1368038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.786+0000 7f1380bc2700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f1368036640 0x7f1368038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f136c009990 tx=0x7f136c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:54.790+0000 7f1371ffb700 1 -- 172.21.15.79:0/3630418187 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1378018b40 con 0x7f137c078ce0 2024-01-19T01:14:56.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.005+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f137c0008d0 con 0x7f1368036640 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.009+0000 7f1371ffb700 1 -- 172.21.15.79:0/3630418187 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f137c0008d0 con 0x7f1368036640 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f1368036640 msgr2=0x7f1368038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f1368036640 0x7f1368038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f136c009990 tx=0x7f136c006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 msgr2=0x7f137c113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c113330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13780094c0 tx=0x7f1378009250 comp rx=0 tx=0).stop 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 shutdown_connections 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f1368036640 0x7f1368038af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 --2- 172.21.15.79:0/3630418187 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f137c078ce0 0x7f137c113330 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 >> 172.21.15.79:0/3630418187 conn(0x7f137c0fda10 msgr2=0x7f137c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 shutdown_connections 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:55.011+0000 7f1383627700 1 -- 172.21.15.79:0/3630418187 wait complete. 2024-01-19T01:14:56.554 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying node-exporter service with default placement... 2024-01-19T01:14:56.861 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:56 smithi079 ceph-mon[28374]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:56.861 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:56 smithi079 ceph-mon[28374]: Saving service grafana spec with placement count:1 2024-01-19T01:14:57.733 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.965+0000 7fa3ce1ac700 1 Processor -- start 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.965+0000 7fa3ce1ac700 1 -- start start 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.965+0000 7fa3ce1ac700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8102be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.965+0000 7fa3ce1ac700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c8103120 con 0x7fa3c81027d0 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8102be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8102be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58714/0 (socket says 172.21.15.79:58714) 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c77fe700 1 -- 172.21.15.79:0/2983225645 learned_addr learned my addr 172.21.15.79:0/2983225645 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c77fe700 1 -- 172.21.15.79:0/2983225645 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8103260 con 0x7fa3c81027d0 2024-01-19T01:14:57.739 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c77fe700 1 --2- 172.21.15.79:0/2983225645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8102be0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80098e0 tx=0x7fa3b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fccc2e038ed50a server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c67fc700 1 -- 172.21.15.79:0/2983225645 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3b800b860 con 0x7fa3c81027d0 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c67fc700 1 -- 172.21.15.79:0/2983225645 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3b800be70 con 0x7fa3c81027d0 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.966+0000 7fa3c67fc700 1 -- 172.21.15.79:0/2983225645 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3b8010380 con 0x7fa3c81027d0 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.967+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/2983225645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 msgr2=0x7fa3c8102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.967+0000 7fa3ce1ac700 1 --2- 172.21.15.79:0/2983225645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8102be0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80098e0 tx=0x7fa3b8009250 comp rx=0 tx=0).stop 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.967+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/2983225645 shutdown_connections 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.967+0000 7fa3ce1ac700 1 --2- 172.21.15.79:0/2983225645 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8102be0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.967+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/2983225645 >> 172.21.15.79:0/2983225645 conn(0x7fa3c80fc0b0 msgr2=0x7fa3c80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.967+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/2983225645 shutdown_connections 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.967+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/2983225645 wait complete. 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3ce1ac700 1 Processor -- start 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3ce1ac700 1 -- start start 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3ce1ac700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:57.740 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3ce1ac700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81010d0 con 0x7fa3c81027d0 2024-01-19T01:14:57.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:57.741 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58726/0 (socket says 172.21.15.79:58726) 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3c77fe700 1 -- 172.21.15.79:0/1392191163 learned_addr learned my addr 172.21.15.79:0/1392191163 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.968+0000 7fa3c77fe700 1 -- 172.21.15.79:0/1392191163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b80094f0 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.969+0000 7fa3c77fe700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8113e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8011b60 tx=0x7fa3b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.969+0000 7fa3affff700 1 -- 172.21.15.79:0/1392191163 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3b8010850 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.969+0000 7fa3affff700 1 -- 172.21.15.79:0/1392191163 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3b8010e60 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.969+0000 7fa3affff700 1 -- 172.21.15.79:0/1392191163 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3b801ae20 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.969+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c81012d0 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.969+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c8114560 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.970+0000 7fa3affff700 1 -- 172.21.15.79:0/1392191163 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa3b802d3f0 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.970+0000 7fa3affff700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fa3b0036640 0x7fa3b0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.970+0000 7fa3affff700 1 -- 172.21.15.79:0/1392191163 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa3b804a1d0 con 0x7fa3c81027d0 2024-01-19T01:14:57.743 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.970+0000 7fa3c6ffd700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fa3b0036640 0x7fa3b0038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.970+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c810a0e0 con 0x7fa3c81027d0 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.970+0000 7fa3c6ffd700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fa3b0036640 0x7fa3b0038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009990 tx=0x7fa3bc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:56.975+0000 7fa3affff700 1 -- 172.21.15.79:0/1392191163 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa3b802b8e0 con 0x7fa3c81027d0 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.166+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3c802b150 con 0x7fa3b0036640 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.171+0000 7fa3affff700 1 -- 172.21.15.79:0/1392191163 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fa3c802b150 con 0x7fa3b0036640 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fa3b0036640 msgr2=0x7fa3b0038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fa3b0036640 0x7fa3b0038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009990 tx=0x7fa3bc006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 msgr2=0x7fa3c8113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8113e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8011b60 tx=0x7fa3b8009250 comp rx=0 tx=0).stop 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 shutdown_connections 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fa3b0036640 0x7fa3b0038af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 --2- 172.21.15.79:0/1392191163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa3c81027d0 0x7fa3c8113e60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 >> 172.21.15.79:0/1392191163 conn(0x7fa3c80fc0b0 msgr2=0x7fa3c80fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:57.744 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 shutdown_connections 2024-01-19T01:14:57.745 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:57.173+0000 7fa3ce1ac700 1 -- 172.21.15.79:0/1392191163 wait complete. 2024-01-19T01:14:57.745 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying alertmanager service with default placement... 2024-01-19T01:14:58.326 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:58 smithi079 ceph-mon[28374]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:58.326 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:58 smithi079 ceph-mon[28374]: Saving service node-exporter spec with placement * 2024-01-19T01:14:58.326 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:58 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a8542b700 1 Processor -- start 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a8542b700 1 -- start start 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a8542b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a8542b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a801053e0 con 0x7f5a80076c60 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a7effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a7effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58736/0 (socket says 172.21.15.79:58736) 2024-01-19T01:14:58.912 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a7effd700 1 -- 172.21.15.79:0/3143369783 learned_addr learned my addr 172.21.15.79:0/3143369783 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a7effd700 1 -- 172.21.15.79:0/3143369783 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a80105520 con 0x7f5a80076c60 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.120+0000 7f5a7effd700 1 --2- 172.21.15.79:0/3143369783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80104ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a700098e0 tx=0x7f5a70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a97bd8935ee31c server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a7dffb700 1 -- 172.21.15.79:0/3143369783 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a7000b860 con 0x7f5a80076c60 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a7dffb700 1 -- 172.21.15.79:0/3143369783 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a7000be70 con 0x7f5a80076c60 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a7dffb700 1 -- 172.21.15.79:0/3143369783 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a70010380 con 0x7f5a80076c60 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a8542b700 1 -- 172.21.15.79:0/3143369783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 msgr2=0x7f5a80104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a8542b700 1 --2- 172.21.15.79:0/3143369783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80104ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a700098e0 tx=0x7f5a70009250 comp rx=0 tx=0).stop 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a8542b700 1 -- 172.21.15.79:0/3143369783 shutdown_connections 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a8542b700 1 --2- 172.21.15.79:0/3143369783 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80104ea0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a8542b700 1 -- 172.21.15.79:0/3143369783 >> 172.21.15.79:0/3143369783 conn(0x7f5a800fc950 msgr2=0x7f5a800fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a8542b700 1 -- 172.21.15.79:0/3143369783 shutdown_connections 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.121+0000 7f5a8542b700 1 -- 172.21.15.79:0/3143369783 wait complete. 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a8542b700 1 Processor -- start 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a8542b700 1 -- start start 2024-01-19T01:14:58.913 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a8542b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a8542b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a80100da0 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a7effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a7effd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:58752/0 (socket says 172.21.15.79:58752) 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a7effd700 1 -- 172.21.15.79:0/2840542978 learned_addr learned my addr 172.21.15.79:0/2840542978 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.122+0000 7f5a7effd700 1 -- 172.21.15.79:0/2840542978 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700094f0 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.123+0000 7f5a7effd700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80102920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a700094c0 tx=0x7f5a70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.123+0000 7f5a677fe700 1 -- 172.21.15.79:0/2840542978 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a70010850 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.123+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a80100fa0 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.123+0000 7f5a677fe700 1 -- 172.21.15.79:0/2840542978 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a70010e60 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.123+0000 7f5a677fe700 1 -- 172.21.15.79:0/2840542978 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a7001a430 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.123+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a801013b0 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.124+0000 7f5a677fe700 1 -- 172.21.15.79:0/2840542978 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5a7001a6b0 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.124+0000 7f5a677fe700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5a680365f0 0x7f5a68038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.124+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8004bb80 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.124+0000 7f5a677fe700 1 -- 172.21.15.79:0/2840542978 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5a7004a440 con 0x7f5a80076c60 2024-01-19T01:14:58.914 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.124+0000 7f5a7e7fc700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5a680365f0 0x7f5a68038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.124+0000 7f5a7e7fc700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5a680365f0 0x7f5a68038aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009990 tx=0x7f5a74006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.128+0000 7f5a677fe700 1 -- 172.21.15.79:0/2840542978 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5a70022340 con 0x7f5a80076c60 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.325+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a8010adf0 con 0x7f5a680365f0 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.329+0000 7f5a677fe700 1 -- 172.21.15.79:0/2840542978 <== mgr.14116 v2:172.21.15.79:6800/3074418411 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f5a8010adf0 con 0x7f5a680365f0 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5a680365f0 msgr2=0x7f5a68038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5a680365f0 0x7f5a68038aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5a74009990 tx=0x7f5a74006e40 comp rx=0 tx=0).stop 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 msgr2=0x7f5a80102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80102920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a700094c0 tx=0x7f5a70009250 comp rx=0 tx=0).stop 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 shutdown_connections 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f5a680365f0 0x7f5a68038aa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 --2- 172.21.15.79:0/2840542978 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5a80076c60 0x7f5a80102920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 >> 172.21.15.79:0/2840542978 conn(0x7f5a800fc950 msgr2=0x7f5a800fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 shutdown_connections 2024-01-19T01:14:58.915 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:58.331+0000 7f5a8542b700 1 -- 172.21.15.79:0/2840542978 wait complete. 2024-01-19T01:14:59.495 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:59 smithi079 ceph-mon[28374]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:14:59.495 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:59 smithi079 ceph-mon[28374]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:14:59.495 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:59 smithi079 ceph-mon[28374]: Saving service alertmanager spec with placement count:1 2024-01-19T01:14:59.495 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:14:59 smithi079 ceph-mon[28374]: from='mgr.14116 172.21.15.79:0/1068163110' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:00.118 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.300+0000 7fcafefff700 1 Processor -- start 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.300+0000 7fcafefff700 1 -- start start 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcafefff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf81026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcafefff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf8102c20 con 0x7fcaf81022d0 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcafcd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf81026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcafcd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf81026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54740/0 (socket says 172.21.15.79:54740) 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcafcd9b700 1 -- 172.21.15.79:0/2311432024 learned_addr learned my addr 172.21.15.79:0/2311432024 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcafcd9b700 1 -- 172.21.15.79:0/2311432024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf8102d60 con 0x7fcaf81022d0 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcafcd9b700 1 --2- 172.21.15.79:0/2311432024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf81026e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0098e0 tx=0x7fcaec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8a85269831466e server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcaf77fe700 1 -- 172.21.15.79:0/2311432024 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaec00b860 con 0x7fcaf81022d0 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcaf77fe700 1 -- 172.21.15.79:0/2311432024 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaec00be70 con 0x7fcaf81022d0 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.301+0000 7fcaf77fe700 1 -- 172.21.15.79:0/2311432024 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaec010380 con 0x7fcaf81022d0 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.302+0000 7fcafefff700 1 -- 172.21.15.79:0/2311432024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 msgr2=0x7fcaf81026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:00.119 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.302+0000 7fcafefff700 1 --2- 172.21.15.79:0/2311432024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf81026e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0098e0 tx=0x7fcaec009250 comp rx=0 tx=0).stop 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.302+0000 7fcafefff700 1 -- 172.21.15.79:0/2311432024 shutdown_connections 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.302+0000 7fcafefff700 1 --2- 172.21.15.79:0/2311432024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf81026e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.302+0000 7fcafefff700 1 -- 172.21.15.79:0/2311432024 >> 172.21.15.79:0/2311432024 conn(0x7fcaf80fda50 msgr2=0x7fcaf80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.302+0000 7fcafefff700 1 -- 172.21.15.79:0/2311432024 shutdown_connections 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.302+0000 7fcafefff700 1 -- 172.21.15.79:0/2311432024 wait complete. 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafefff700 1 Processor -- start 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafefff700 1 -- start start 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafefff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf8111170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafefff700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf81116b0 con 0x7fcaf81022d0 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafcd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf8111170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafcd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf8111170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54748/0 (socket says 172.21.15.79:54748) 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafcd9b700 1 -- 172.21.15.79:0/1535938577 learned_addr learned my addr 172.21.15.79:0/1535938577 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafcd9b700 1 -- 172.21.15.79:0/1535938577 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaec0094f0 con 0x7fcaf81022d0 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.303+0000 7fcafcd9b700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf8111170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcaec011b60 tx=0x7fcaec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:00.120 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.304+0000 7fcaf57fa700 1 -- 172.21.15.79:0/1535938577 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaec010850 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.304+0000 7fcaf57fa700 1 -- 172.21.15.79:0/1535938577 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaec010e60 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.304+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf81118b0 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.304+0000 7fcaf57fa700 1 -- 172.21.15.79:0/1535938577 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaec01ae20 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.304+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf8111cc0 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.304+0000 7fcaf57fa700 1 -- 172.21.15.79:0/1535938577 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fcaec02d3f0 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.305+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf804bb80 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.305+0000 7fcaf57fa700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fcae8036640 0x7fcae8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.305+0000 7fcaf57fa700 1 -- 172.21.15.79:0/1535938577 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcaec04a3c0 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.305+0000 7fcaf7fff700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fcae8036640 0x7fcae8038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.305+0000 7fcaf7fff700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fcae8036640 0x7fcae8038af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009990 tx=0x7fcae4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.309+0000 7fcaf57fa700 1 -- 172.21.15.79:0/1535938577 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcaec02b8e0 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.493+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fcaf81040b0 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.499+0000 7fcaf57fa700 1 -- 172.21.15.79:0/1535938577 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fcaec02b2f0 con 0x7fcaf81022d0 2024-01-19T01:15:00.121 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fcae8036640 msgr2=0x7fcae8038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fcae8036640 0x7fcae8038af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009990 tx=0x7fcae4006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 msgr2=0x7fcaf8111170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf8111170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcaec011b60 tx=0x7fcaec009250 comp rx=0 tx=0).stop 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 shutdown_connections 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fcae8036640 0x7fcae8038af0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 --2- 172.21.15.79:0/1535938577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcaf81022d0 0x7fcaf8111170 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 >> 172.21.15.79:0/1535938577 conn(0x7fcaf80fda50 msgr2=0x7fcaf80ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 shutdown_connections 2024-01-19T01:15:00.122 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:14:59.501+0000 7fcafefff700 1 -- 172.21.15.79:0/1535938577 wait complete. 2024-01-19T01:15:00.747 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:00 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1535938577' entity='client.admin' 2024-01-19T01:15:01.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.555+0000 7fb046067700 1 Processor -- start 2024-01-19T01:15:01.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.555+0000 7fb046067700 1 -- start start 2024-01-19T01:15:01.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.555+0000 7fb046067700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:01.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.555+0000 7fb046067700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040104310 con 0x7fb0401039c0 2024-01-19T01:15:01.234 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.555+0000 7fb03f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.555+0000 7fb03f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54750/0 (socket says 172.21.15.79:54750) 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.555+0000 7fb03f7fe700 1 -- 172.21.15.79:0/174129652 learned_addr learned my addr 172.21.15.79:0/174129652 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.556+0000 7fb03f7fe700 1 -- 172.21.15.79:0/174129652 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb040101fb0 con 0x7fb0401039c0 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.556+0000 7fb03f7fe700 1 --2- 172.21.15.79:0/174129652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040103dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098e0 tx=0x7fb030009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c37ff49d00dc18 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.556+0000 7fb03effd700 1 -- 172.21.15.79:0/174129652 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb03000b860 con 0x7fb0401039c0 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.556+0000 7fb03effd700 1 -- 172.21.15.79:0/174129652 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb03000be70 con 0x7fb0401039c0 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.556+0000 7fb03effd700 1 -- 172.21.15.79:0/174129652 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030010380 con 0x7fb0401039c0 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.556+0000 7fb046067700 1 -- 172.21.15.79:0/174129652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 msgr2=0x7fb040103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.556+0000 7fb046067700 1 --2- 172.21.15.79:0/174129652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040103dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098e0 tx=0x7fb030009250 comp rx=0 tx=0).stop 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 -- 172.21.15.79:0/174129652 shutdown_connections 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 --2- 172.21.15.79:0/174129652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040103dd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 -- 172.21.15.79:0/174129652 >> 172.21.15.79:0/174129652 conn(0x7fb0400fda40 msgr2=0x7fb0400ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 -- 172.21.15.79:0/174129652 shutdown_connections 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 -- 172.21.15.79:0/174129652 wait complete. 2024-01-19T01:15:01.235 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 Processor -- start 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 -- start start 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb046067700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040111670 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.557+0000 7fb03f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb03f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54758/0 (socket says 172.21.15.79:54758) 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb03f7fe700 1 -- 172.21.15.79:0/626975821 learned_addr learned my addr 172.21.15.79:0/626975821 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb03f7fe700 1 -- 172.21.15.79:0/626975821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300094f0 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb03f7fe700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040111130 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb030003080 tx=0x7fb030009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb03cff9700 1 -- 172.21.15.79:0/626975821 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030010800 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb03cff9700 1 -- 172.21.15.79:0/626975821 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb030010e10 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb040111870 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb03cff9700 1 -- 172.21.15.79:0/626975821 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb03001a430 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.558+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb040111c80 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.559+0000 7fb03cff9700 1 -- 172.21.15.79:0/626975821 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb03001a660 con 0x7fb0401039c0 2024-01-19T01:15:01.236 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.559+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb040102d30 con 0x7fb0401039c0 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.559+0000 7fb03cff9700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb02c036640 0x7fb02c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.559+0000 7fb03cff9700 1 -- 172.21.15.79:0/626975821 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb03004a5a0 con 0x7fb0401039c0 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.559+0000 7fb036ffd700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb02c036640 0x7fb02c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.560+0000 7fb036ffd700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb02c036640 0x7fb02c038af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb028009990 tx=0x7fb028006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.564+0000 7fb03cff9700 1 -- 172.21.15.79:0/626975821 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb030018b40 con 0x7fb0401039c0 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.746+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb04004bb80 con 0x7fb0401039c0 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.750+0000 7fb03cff9700 1 -- 172.21.15.79:0/626975821 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb030018b40 con 0x7fb0401039c0 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb02c036640 msgr2=0x7fb02c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb02c036640 0x7fb02c038af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb028009990 tx=0x7fb028006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 msgr2=0x7fb040111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040111130 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb030003080 tx=0x7fb030009250 comp rx=0 tx=0).stop 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 shutdown_connections 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb02c036640 0x7fb02c038af0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 --2- 172.21.15.79:0/626975821 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb0401039c0 0x7fb040111130 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:01.237 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.752+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 >> 172.21.15.79:0/626975821 conn(0x7fb0400fda40 msgr2=0x7fb0400ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:01.238 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.753+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 shutdown_connections 2024-01-19T01:15:01.238 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:00.753+0000 7fb046067700 1 -- 172.21.15.79:0/626975821 wait complete. 2024-01-19T01:15:01.238 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling the dashboard module... 2024-01-19T01:15:01.968 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:01 smithi079 ceph-mon[28374]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:01.968 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:01 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/626975821' entity='client.admin' 2024-01-19T01:15:02.890 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:02 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2288387020' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0bb03700 1 Processor -- start 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0bb03700 1 -- start start 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0bb03700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0bb03700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04104310 con 0x7f0e041039c0 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0989f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0989f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54764/0 (socket says 172.21.15.79:54764) 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0989f700 1 -- 172.21.15.79:0/3735054941 learned_addr learned my addr 172.21.15.79:0/3735054941 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.724+0000 7f0e0989f700 1 -- 172.21.15.79:0/3735054941 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04101fb0 con 0x7f0e041039c0 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0989f700 1 --2- 172.21.15.79:0/3735054941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04103dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e000098e0 tx=0x7f0e00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d91a503738bd3ad server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0889d700 1 -- 172.21.15.79:0/3735054941 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e0000b860 con 0x7f0e041039c0 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0889d700 1 -- 172.21.15.79:0/3735054941 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e0000be70 con 0x7f0e041039c0 2024-01-19T01:15:03.166 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0889d700 1 -- 172.21.15.79:0/3735054941 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e00010380 con 0x7f0e041039c0 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0bb03700 1 -- 172.21.15.79:0/3735054941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 msgr2=0x7f0e04103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0bb03700 1 --2- 172.21.15.79:0/3735054941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04103dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e000098e0 tx=0x7f0e00009250 comp rx=0 tx=0).stop 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0bb03700 1 -- 172.21.15.79:0/3735054941 shutdown_connections 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0bb03700 1 --2- 172.21.15.79:0/3735054941 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04103dd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0bb03700 1 -- 172.21.15.79:0/3735054941 >> 172.21.15.79:0/3735054941 conn(0x7f0e040fda40 msgr2=0x7f0e040ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0bb03700 1 -- 172.21.15.79:0/3735054941 shutdown_connections 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.725+0000 7f0e0bb03700 1 -- 172.21.15.79:0/3735054941 wait complete. 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0bb03700 1 Processor -- start 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0bb03700 1 -- start start 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0bb03700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0bb03700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04111670 con 0x7f0e041039c0 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0989f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0989f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54778/0 (socket says 172.21.15.79:54778) 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0989f700 1 -- 172.21.15.79:0/2288387020 learned_addr learned my addr 172.21.15.79:0/2288387020 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.726+0000 7f0e0989f700 1 -- 172.21.15.79:0/2288387020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e000094f0 con 0x7f0e041039c0 2024-01-19T01:15:03.167 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.727+0000 7f0e0989f700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04111130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000bac0 tx=0x7f0e00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.727+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e000109b0 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.727+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e0001a430 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.727+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e04111870 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.727+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e000245b0 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.727+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e04111c80 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.728+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f0e00024820 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.728+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e04102d30 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.728+0000 7f0dfa7fc700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f0df00365f0 0x7f0df0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.728+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0e0004a720 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.728+0000 7f0e0909e700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f0df00365f0 0x7f0df0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.728+0000 7f0e0909e700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f0df00365f0 0x7f0df0038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009990 tx=0x7f0df4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.733+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0e00018940 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:01.967+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0e0404bb80 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.507+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f0e0002e460 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.508+0000 7f0dfa7fc700 1 -- 172.21.15.79:0/2288387020 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0e00023350 con 0x7f0e041039c0 2024-01-19T01:15:03.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f0df00365f0 msgr2=0x7f0df0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f0df00365f0 0x7f0df0038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009990 tx=0x7f0df4006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 msgr2=0x7f0e04111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04111130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000bac0 tx=0x7f0e00009250 comp rx=0 tx=0).stop 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 shutdown_connections 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f0df00365f0 0x7f0df0038aa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 --2- 172.21.15.79:0/2288387020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0e041039c0 0x7f0e04111130 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 >> 172.21.15.79:0/2288387020 conn(0x7f0e040fda40 msgr2=0x7f0e040ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 shutdown_connections 2024-01-19T01:15:03.169 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:02.511+0000 7f0e0bb03700 1 -- 172.21.15.79:0/2288387020 wait complete. 2024-01-19T01:15:03.746 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:03 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2288387020' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-19T01:15:03.746 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:03 smithi079 ceph-mon[28374]: mgrmap e9: smithi079.miqczk(active, since 24s) 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "smithi079.miqczk", 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.539+0000 7fb1fbdd6700 1 Processor -- start 2024-01-19T01:15:04.373 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.539+0000 7fb1fbdd6700 1 -- start start 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1fbdd6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f4102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1fbdd6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f4103250 con 0x7fb1f4102900 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f9b72700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f4102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f9b72700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f4102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54802/0 (socket says 172.21.15.79:54802) 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f9b72700 1 -- 172.21.15.79:0/4205742840 learned_addr learned my addr 172.21.15.79:0/4205742840 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f9b72700 1 -- 172.21.15.79:0/4205742840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f4103390 con 0x7fb1f4102900 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f9b72700 1 --2- 172.21.15.79:0/4205742840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f4102d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00098e0 tx=0x7fb1f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2c8073549afab6 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f8b70700 1 -- 172.21.15.79:0/4205742840 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f000b860 con 0x7fb1f4102900 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f8b70700 1 -- 172.21.15.79:0/4205742840 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1f000be70 con 0x7fb1f4102900 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.540+0000 7fb1f8b70700 1 -- 172.21.15.79:0/4205742840 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f0010380 con 0x7fb1f4102900 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.541+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/4205742840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 msgr2=0x7fb1f4102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.541+0000 7fb1fbdd6700 1 --2- 172.21.15.79:0/4205742840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f4102d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00098e0 tx=0x7fb1f0009250 comp rx=0 tx=0).stop 2024-01-19T01:15:04.374 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.541+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/4205742840 shutdown_connections 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.541+0000 7fb1fbdd6700 1 --2- 172.21.15.79:0/4205742840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f4102d10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.541+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/4205742840 >> 172.21.15.79:0/4205742840 conn(0x7fb1f40fc0b0 msgr2=0x7fb1f40fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.541+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/4205742840 shutdown_connections 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.541+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/4205742840 wait complete. 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1fbdd6700 1 Processor -- start 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1fbdd6700 1 -- start start 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1fbdd6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f41133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1fbdd6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41138f0 con 0x7fb1f4102900 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1f9b72700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f41133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1f9b72700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f41133b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54804/0 (socket says 172.21.15.79:54804) 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1f9b72700 1 -- 172.21.15.79:0/303444318 learned_addr learned my addr 172.21.15.79:0/303444318 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1f9b72700 1 -- 172.21.15.79:0/303444318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f00094f0 con 0x7fb1f4102900 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1f9b72700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f41133b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000c010 tx=0x7fb1f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:04.375 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1ea7fc700 1 -- 172.21.15.79:0/303444318 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f0010960 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1ea7fc700 1 -- 172.21.15.79:0/303444318 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1f001a430 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.542+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f4113af0 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.543+0000 7fb1ea7fc700 1 -- 172.21.15.79:0/303444318 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1f0019580 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.543+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f4113f00 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.543+0000 7fb1ea7fc700 1 -- 172.21.15.79:0/303444318 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fb1f00197b0 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.544+0000 7fb1ea7fc700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 0x7fb1e0038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.544+0000 7fb1ea7fc700 1 -- 172.21.15.79:0/303444318 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb1f004a5c0 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.544+0000 7fb1f9371700 1 -- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 msgr2=0x7fb1e0038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3074418411 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.544+0000 7fb1f9371700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 0x7fb1e0038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.544+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f404bb80 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.548+0000 7fb1ea7fc700 1 -- 172.21.15.79:0/303444318 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb1f002bb80 con 0x7fb1f4102900 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.745+0000 7fb1f9371700 1 -- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 msgr2=0x7fb1e0038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3074418411 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.745+0000 7fb1f9371700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 0x7fb1e0038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T01:15:04.376 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.795+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb1f410d100 con 0x7fb1f4102900 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.795+0000 7fb1ea7fc700 1 -- 172.21.15.79:0/303444318 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb1f002b770 con 0x7fb1f4102900 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 msgr2=0x7fb1e0038b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 0x7fb1e0038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 msgr2=0x7fb1f41133b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f41133b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000c010 tx=0x7fb1f0009250 comp rx=0 tx=0).stop 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 shutdown_connections 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7fb1e0036690 0x7fb1e0038b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 --2- 172.21.15.79:0/303444318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1f4102900 0x7fb1f41133b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 >> 172.21.15.79:0/303444318 conn(0x7fb1f40fc0b0 msgr2=0x7fb1f40fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 shutdown_connections 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:03.797+0000 7fb1fbdd6700 1 -- 172.21.15.79:0/303444318 wait complete. 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-01-19T01:15:04.377 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 9... 2024-01-19T01:15:04.637 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:04 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/303444318' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: Active manager daemon smithi079.miqczk restarted 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: Activating manager daemon smithi079.miqczk 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: osdmap e3: 0 total, 0 up, 0 in 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: mgrmap e10: smithi079.miqczk(active, starting, since 0.00316465s) 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr metadata", "who": "smithi079.miqczk", "id": "smithi079.miqczk"}]: dispatch 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: Manager daemon smithi079.miqczk is now available 2024-01-19T01:15:09.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:15:09.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:09 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:15:10.457 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:10 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/mirror_snapshot_schedule"}]: dispatch 2024-01-19T01:15:10.457 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:10 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/trash_purge_schedule"}]: dispatch 2024-01-19T01:15:10.457 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:10 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:10.457 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:10 smithi079 ceph-mon[28374]: mgrmap e11: smithi079.miqczk(active, since 1.00593s) 2024-01-19T01:15:10.758 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-01-19T01:15:10.758 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-01-19T01:15:10.758 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-19T01:15:10.758 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-01-19T01:15:10.758 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.849+0000 7f907b897700 1 Processor -- start 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.849+0000 7f907b897700 1 -- start start 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.849+0000 7f907b897700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f90741026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.849+0000 7f907b897700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074102c00 con 0x7f90741022b0 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.849+0000 7f9079633700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f90741026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.849+0000 7f9079633700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f90741026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54820/0 (socket says 172.21.15.79:54820) 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.849+0000 7f9079633700 1 -- 172.21.15.79:0/1872868644 learned_addr learned my addr 172.21.15.79:0/1872868644 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.850+0000 7f9079633700 1 -- 172.21.15.79:0/1872868644 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074102d40 con 0x7f90741022b0 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.850+0000 7f9079633700 1 --2- 172.21.15.79:0/1872868644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f90741026c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90700098e0 tx=0x7f9070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fb44d922dfab39 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.850+0000 7f906bfff700 1 -- 172.21.15.79:0/1872868644 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f907000b860 con 0x7f90741022b0 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.850+0000 7f906bfff700 1 -- 172.21.15.79:0/1872868644 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f907000be70 con 0x7f90741022b0 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.850+0000 7f906bfff700 1 -- 172.21.15.79:0/1872868644 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9070010380 con 0x7f90741022b0 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.851+0000 7f907b897700 1 -- 172.21.15.79:0/1872868644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 msgr2=0x7f90741026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.851+0000 7f907b897700 1 --2- 172.21.15.79:0/1872868644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f90741026c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90700098e0 tx=0x7f9070009250 comp rx=0 tx=0).stop 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.851+0000 7f907b897700 1 -- 172.21.15.79:0/1872868644 shutdown_connections 2024-01-19T01:15:10.759 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.851+0000 7f907b897700 1 --2- 172.21.15.79:0/1872868644 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f90741026c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.851+0000 7f907b897700 1 -- 172.21.15.79:0/1872868644 >> 172.21.15.79:0/1872868644 conn(0x7f90740fda10 msgr2=0x7f90740ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.851+0000 7f907b897700 1 -- 172.21.15.79:0/1872868644 shutdown_connections 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.851+0000 7f907b897700 1 -- 172.21.15.79:0/1872868644 wait complete. 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f907b897700 1 Processor -- start 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f907b897700 1 -- start start 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f907b897700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f9074111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f907b897700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90741116d0 con 0x7f90741022b0 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f9079633700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f9074111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f9079633700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f9074111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54822/0 (socket says 172.21.15.79:54822) 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f9079633700 1 -- 172.21.15.79:0/3484381906 learned_addr learned my addr 172.21.15.79:0/3484381906 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f9079633700 1 -- 172.21.15.79:0/3484381906 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700094f0 con 0x7f90741022b0 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.852+0000 7f9079633700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f9074111190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f907000c010 tx=0x7f9070009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.853+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9070010960 con 0x7f90741022b0 2024-01-19T01:15:10.760 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.853+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90741118d0 con 0x7f90741022b0 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.853+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f907001a430 con 0x7f90741022b0 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.853+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9070019580 con 0x7f90741022b0 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.853+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9074111ce0 con 0x7f90741022b0 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.854+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f90700197b0 con 0x7f90741022b0 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.854+0000 7f9069ffb700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.854+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 --> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f907404bb80 con 0x7f9060036690 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.854+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f907001a980 con 0x7f90741022b0 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.854+0000 7f9078e32700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 msgr2=0x7f9060038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3074418411 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:04.854+0000 7f9078e32700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:05.055+0000 7f9078e32700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 msgr2=0x7f9060038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3074418411 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:05.055+0000 7f9078e32700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:05.456+0000 7f9078e32700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 msgr2=0x7f9060038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3074418411 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:05.456+0000 7f9078e32700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-19T01:15:10.761 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:06.257+0000 7f9078e32700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 msgr2=0x7f9060038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3074418411 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:06.257+0000 7f9078e32700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:07.858+0000 7f9078e32700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 msgr2=0x7f9060038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/3074418411 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:07.858+0000 7f9078e32700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:09.095+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mgrmap(e 10) v1 ==== 43061+0+0 (secure 0 0 0) 0x7f9070019d30 con 0x7f90741022b0 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:09.095+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 msgr2=0x7f9060038b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:09.095+0000 7f9069ffb700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.097+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f907004a920 con 0x7f90741022b0 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.097+0000 7f9069ffb700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f9060039f90 0x7f906003c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.097+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f907404bb80 con 0x7f9060039f90 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.097+0000 7f9078e32700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f9060039f90 0x7f906003c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.098+0000 7f9078e32700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f9060039f90 0x7f906003c370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f90640039d0 tx=0x7f9064009350 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.098+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mgr.14158 v2:172.21.15.79:6800/4076006582 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f907404bb80 con 0x7f9060039f90 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.104+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9074104090 con 0x7f9060039f90 2024-01-19T01:15:10.762 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.104+0000 7f9069ffb700 1 -- 172.21.15.79:0/3484381906 <== mgr.14158 v2:172.21.15.79:6800/4076006582 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f9074104090 con 0x7f9060039f90 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f9060039f90 msgr2=0x7f906003c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f9060039f90 0x7f906003c370 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f90640039d0 tx=0x7f9064009350 comp rx=0 tx=0).stop 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 msgr2=0x7f9074111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f9074111190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f907000c010 tx=0x7f9070009250 comp rx=0 tx=0).stop 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 shutdown_connections 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f9060039f90 0x7f906003c370 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f90741022b0 0x7f9074111190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 --2- 172.21.15.79:0/3484381906 >> [v2:172.21.15.79:6800/3074418411,v1:172.21.15.79:6801/3074418411] conn(0x7f9060036690 0x7f9060038b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 >> 172.21.15.79:0/3484381906 conn(0x7f90740fda10 msgr2=0x7f90740ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 shutdown_connections 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:10.105+0000 7f907b897700 1 -- 172.21.15.79:0/3484381906 wait complete. 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 9 is available 2024-01-19T01:15:10.763 INFO:teuthology.orchestra.run.smithi079.stdout:Generating a dashboard self-signed certificate... 2024-01-19T01:15:11.721 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:11 smithi079 ceph-mon[28374]: [19/Jan/2024:01:15:09] ENGINE Bus STARTING 2024-01-19T01:15:11.721 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:11 smithi079 ceph-mon[28374]: [19/Jan/2024:01:15:09] ENGINE Serving on https://172.21.15.79:7150 2024-01-19T01:15:11.721 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:11 smithi079 ceph-mon[28374]: [19/Jan/2024:01:15:09] ENGINE Bus STARTED 2024-01-19T01:15:11.721 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:11 smithi079 ceph-mon[28374]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-19T01:15:11.722 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:11 smithi079 ceph-mon[28374]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-19T01:15:11.722 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:11 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:11.722 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:11 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:12.391 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-01-19T01:15:12.391 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.182+0000 7fdd5b3c6700 1 Processor -- start 2024-01-19T01:15:12.391 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.182+0000 7fdd5b3c6700 1 -- start start 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.182+0000 7fdd5b3c6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd54104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.182+0000 7fdd5b3c6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54105420 con 0x7fdd54070fc0 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd59162700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd54104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd59162700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd54104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36302/0 (socket says 172.21.15.79:36302) 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd59162700 1 -- 172.21.15.79:0/763457675 learned_addr learned my addr 172.21.15.79:0/763457675 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd59162700 1 -- 172.21.15.79:0/763457675 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd54105560 con 0x7fdd54070fc0 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd59162700 1 --2- 172.21.15.79:0/763457675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd54104ee0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd500098e0 tx=0x7fdd50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ccd72ce753eec7 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd4bfff700 1 -- 172.21.15.79:0/763457675 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd5000b860 con 0x7fdd54070fc0 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd4bfff700 1 -- 172.21.15.79:0/763457675 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd5000be70 con 0x7fdd54070fc0 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.183+0000 7fdd4bfff700 1 -- 172.21.15.79:0/763457675 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd50010380 con 0x7fdd54070fc0 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/763457675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 msgr2=0x7fdd54104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 --2- 172.21.15.79:0/763457675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd54104ee0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd500098e0 tx=0x7fdd50009250 comp rx=0 tx=0).stop 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/763457675 shutdown_connections 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 --2- 172.21.15.79:0/763457675 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd54104ee0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:12.392 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/763457675 >> 172.21.15.79:0/763457675 conn(0x7fdd540fc970 msgr2=0x7fdd540fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/763457675 shutdown_connections 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/763457675 wait complete. 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.184+0000 7fdd5b3c6700 1 Processor -- start 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd5b3c6700 1 -- start start 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd5b3c6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd541111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd5b3c6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd54111700 con 0x7fdd54070fc0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd59162700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd541111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd59162700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd541111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36308/0 (socket says 172.21.15.79:36308) 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd59162700 1 -- 172.21.15.79:0/1980348126 learned_addr learned my addr 172.21.15.79:0/1980348126 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd59162700 1 -- 172.21.15.79:0/1980348126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500094f0 con 0x7fdd54070fc0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd59162700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd541111c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdd50006d40 tx=0x7fdd50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd50010960 con 0x7fdd54070fc0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd5001a430 con 0x7fdd54070fc0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd500245b0 con 0x7fdd54070fc0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.185+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd54111900 con 0x7fdd54070fc0 2024-01-19T01:15:12.393 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.186+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd54111d10 con 0x7fdd54070fc0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.186+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7fdd5002d3f0 con 0x7fdd54070fc0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.186+0000 7fdd49ffb700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fdd40036510 0x7fdd400389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.186+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdd5004a200 con 0x7fdd54070fc0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.187+0000 7fdd58961700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fdd40036510 0x7fdd400389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.187+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5404bb80 con 0x7fdd54070fc0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.187+0000 7fdd58961700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fdd40036510 0x7fdd400389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009990 tx=0x7fdd44006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.191+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdd50018d20 con 0x7fdd54070fc0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.412+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fdd5001a9f0 con 0x7fdd54070fc0 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.419+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd5410ad10 con 0x7fdd40036510 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.611+0000 7fdd49ffb700 1 -- 172.21.15.79:0/1980348126 <== mgr.14158 v2:172.21.15.79:6800/4076006582 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fdd5410ad10 con 0x7fdd40036510 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fdd40036510 msgr2=0x7fdd400389c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fdd40036510 0x7fdd400389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd44009990 tx=0x7fdd44006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 msgr2=0x7fdd541111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:12.394 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd541111c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdd50006d40 tx=0x7fdd50009250 comp rx=0 tx=0).stop 2024-01-19T01:15:12.395 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 shutdown_connections 2024-01-19T01:15:12.395 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fdd40036510 0x7fdd400389c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:12.395 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 --2- 172.21.15.79:0/1980348126 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdd54070fc0 0x7fdd541111c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:12.395 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 >> 172.21.15.79:0/1980348126 conn(0x7fdd540fc970 msgr2=0x7fdd540fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:12.395 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 shutdown_connections 2024-01-19T01:15:12.395 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:11.613+0000 7fdd5b3c6700 1 -- 172.21.15.79:0/1980348126 wait complete. 2024-01-19T01:15:12.395 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial admin user... 2024-01-19T01:15:12.720 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:12 smithi079 ceph-mon[28374]: mgrmap e12: smithi079.miqczk(active, since 2s) 2024-01-19T01:15:12.720 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:12 smithi079 ceph-mon[28374]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:15:12.720 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:12 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:12.720 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:12 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:14.202 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$1GwKPubq7w6ZT3Vtmp1LxeWKHzZ0uOjg06FHxnREcaExxUjPE/tfa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705626913, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.818+0000 7f7c3218e700 1 Processor -- start 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.818+0000 7f7c3218e700 1 -- start start 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.818+0000 7f7c3218e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.818+0000 7f7c3218e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c104310 con 0x7f7c2c1039c0 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.818+0000 7f7c2b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.818+0000 7f7c2b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36320/0 (socket says 172.21.15.79:36320) 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.818+0000 7f7c2b7fe700 1 -- 172.21.15.79:0/1366335740 learned_addr learned my addr 172.21.15.79:0/1366335740 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c2b7fe700 1 -- 172.21.15.79:0/1366335740 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c101fb0 con 0x7f7c2c1039c0 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c2b7fe700 1 --2- 172.21.15.79:0/1366335740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c103dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0098e0 tx=0x7f7c1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9822dcbbd136c72 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c2a7fc700 1 -- 172.21.15.79:0/1366335740 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c00b860 con 0x7f7c2c1039c0 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c2a7fc700 1 -- 172.21.15.79:0/1366335740 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c1c00be70 con 0x7f7c2c1039c0 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c2a7fc700 1 -- 172.21.15.79:0/1366335740 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c010380 con 0x7f7c2c1039c0 2024-01-19T01:15:14.203 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c3218e700 1 -- 172.21.15.79:0/1366335740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 msgr2=0x7f7c2c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c3218e700 1 --2- 172.21.15.79:0/1366335740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c103dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c0098e0 tx=0x7f7c1c009250 comp rx=0 tx=0).stop 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c3218e700 1 -- 172.21.15.79:0/1366335740 shutdown_connections 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c3218e700 1 --2- 172.21.15.79:0/1366335740 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c103dd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c3218e700 1 -- 172.21.15.79:0/1366335740 >> 172.21.15.79:0/1366335740 conn(0x7f7c2c0fda40 msgr2=0x7f7c2c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.819+0000 7f7c3218e700 1 -- 172.21.15.79:0/1366335740 shutdown_connections 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c3218e700 1 -- 172.21.15.79:0/1366335740 wait complete. 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c3218e700 1 Processor -- start 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c3218e700 1 -- start start 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c3218e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c3218e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c077190 con 0x7f7c2c1039c0 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c2b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c2b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36332/0 (socket says 172.21.15.79:36332) 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.820+0000 7f7c2b7fe700 1 -- 172.21.15.79:0/3158550386 learned_addr learned my addr 172.21.15.79:0/3158550386 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:14.204 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.821+0000 7f7c2b7fe700 1 -- 172.21.15.79:0/3158550386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c0094f0 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.821+0000 7f7c2b7fe700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c078d10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c003080 tx=0x7f7c1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.821+0000 7f7c13fff700 1 -- 172.21.15.79:0/3158550386 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c010800 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.821+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c077390 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.821+0000 7f7c13fff700 1 -- 172.21.15.79:0/3158550386 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c1c010e10 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.821+0000 7f7c13fff700 1 -- 172.21.15.79:0/3158550386 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c01a430 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.821+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c0777a0 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.822+0000 7f7c13fff700 1 -- 172.21.15.79:0/3158550386 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f7c1c01a6b0 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.822+0000 7f7c13fff700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7c14036640 0x7f7c14038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.822+0000 7f7c13fff700 1 -- 172.21.15.79:0/3158550386 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7c1c04a780 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.822+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c05f270 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.822+0000 7f7c2affd700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7c14036640 0x7f7c14038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.822+0000 7f7c2affd700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7c14036640 0x7f7c14038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009990 tx=0x7f7c20006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:12.827+0000 7f7c13fff700 1 -- 172.21.15.79:0/3158550386 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7c1c022520 con 0x7f7c2c1039c0 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.035+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7c2c05f090 con 0x7f7c14036640 2024-01-19T01:15:14.205 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.256+0000 7f7c13fff700 1 -- 172.21.15.79:0/3158550386 <== mgr.14158 v2:172.21.15.79:6800/4076006582 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7c2c05f090 con 0x7f7c14036640 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7c14036640 msgr2=0x7f7c14038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7c14036640 0x7f7c14038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009990 tx=0x7f7c20006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 msgr2=0x7f7c2c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c078d10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c003080 tx=0x7f7c1c009250 comp rx=0 tx=0).stop 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 shutdown_connections 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7c14036640 0x7f7c14038af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 --2- 172.21.15.79:0/3158550386 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7c2c1039c0 0x7f7c2c078d10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 >> 172.21.15.79:0/3158550386 conn(0x7f7c2c0fda40 msgr2=0x7f7c2c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 shutdown_connections 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:13.258+0000 7f7c3218e700 1 -- 172.21.15.79:0/3158550386 wait complete. 2024-01-19T01:15:14.206 INFO:teuthology.orchestra.run.smithi079.stdout:Fetching dashboard port number... 2024-01-19T01:15:14.449 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:14 smithi079 ceph-mon[28374]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:15:14.449 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:14 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:15.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:15 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1529138212' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-19T01:15:15.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 8443 2024-01-19T01:15:15.552 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b0a565700 1 Processor -- start 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b0a565700 1 -- start start 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b0a565700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b0a565700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b04104310 con 0x7f1b041039c0 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b03fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b03fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36340/0 (socket says 172.21.15.79:36340) 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b03fff700 1 -- 172.21.15.79:0/2782950415 learned_addr learned my addr 172.21.15.79:0/2782950415 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b03fff700 1 -- 172.21.15.79:0/2782950415 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b04101fb0 con 0x7f1b041039c0 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.668+0000 7f1b03fff700 1 --2- 172.21.15.79:0/2782950415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04103dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1af40098e0 tx=0x7f1af4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b7c5f1876a2336 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b02ffd700 1 -- 172.21.15.79:0/2782950415 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af400b860 con 0x7f1b041039c0 2024-01-19T01:15:15.553 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b02ffd700 1 -- 172.21.15.79:0/2782950415 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1af400be70 con 0x7f1b041039c0 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b02ffd700 1 -- 172.21.15.79:0/2782950415 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af4010380 con 0x7f1b041039c0 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b0a565700 1 -- 172.21.15.79:0/2782950415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 msgr2=0x7f1b04103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b0a565700 1 --2- 172.21.15.79:0/2782950415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04103dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1af40098e0 tx=0x7f1af4009250 comp rx=0 tx=0).stop 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b0a565700 1 -- 172.21.15.79:0/2782950415 shutdown_connections 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b0a565700 1 --2- 172.21.15.79:0/2782950415 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04103dd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b0a565700 1 -- 172.21.15.79:0/2782950415 >> 172.21.15.79:0/2782950415 conn(0x7f1b040fda40 msgr2=0x7f1b040ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b0a565700 1 -- 172.21.15.79:0/2782950415 shutdown_connections 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.669+0000 7f1b0a565700 1 -- 172.21.15.79:0/2782950415 wait complete. 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b0a565700 1 Processor -- start 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b0a565700 1 -- start start 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b0a565700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b0a565700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b04111670 con 0x7f1b041039c0 2024-01-19T01:15:15.554 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b03fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b03fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36348/0 (socket says 172.21.15.79:36348) 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b03fff700 1 -- 172.21.15.79:0/1529138212 learned_addr learned my addr 172.21.15.79:0/1529138212 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b03fff700 1 -- 172.21.15.79:0/1529138212 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af40094f0 con 0x7f1b041039c0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.670+0000 7f1b03fff700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04111130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1af400bac0 tx=0x7f1af4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.671+0000 7f1b00ff9700 1 -- 172.21.15.79:0/1529138212 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af40109b0 con 0x7f1b041039c0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.671+0000 7f1b00ff9700 1 -- 172.21.15.79:0/1529138212 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1af401a430 con 0x7f1b041039c0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.671+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b04111870 con 0x7f1b041039c0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.671+0000 7f1b00ff9700 1 -- 172.21.15.79:0/1529138212 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1af40245b0 con 0x7f1b041039c0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.671+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b04111c80 con 0x7f1b041039c0 2024-01-19T01:15:15.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.671+0000 7f1b00ff9700 1 -- 172.21.15.79:0/1529138212 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f1af4024710 con 0x7f1b041039c0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.672+0000 7f1b00ff9700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f1aec036690 0x7f1aec038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.672+0000 7f1b00ff9700 1 -- 172.21.15.79:0/1529138212 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1af404a6b0 con 0x7f1b041039c0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.672+0000 7f1b037fe700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f1aec036690 0x7f1aec038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.672+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0405f270 con 0x7f1b041039c0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.672+0000 7f1b037fe700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f1aec036690 0x7f1aec038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009990 tx=0x7f1af8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.677+0000 7f1b00ff9700 1 -- 172.21.15.79:0/1529138212 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1af4018940 con 0x7f1b041039c0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.861+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f1b04102d30 con 0x7f1b041039c0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.861+0000 7f1b00ff9700 1 -- 172.21.15.79:0/1529138212 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1af4048030 con 0x7f1b041039c0 2024-01-19T01:15:15.556 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f1aec036690 msgr2=0x7f1aec038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f1aec036690 0x7f1aec038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009990 tx=0x7f1af8006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 msgr2=0x7f1b04111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04111130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1af400bac0 tx=0x7f1af4009250 comp rx=0 tx=0).stop 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 shutdown_connections 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f1aec036690 0x7f1aec038b40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 --2- 172.21.15.79:0/1529138212 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1b041039c0 0x7f1b04111130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 >> 172.21.15.79:0/1529138212 conn(0x7f1b040fda40 msgr2=0x7f1b040ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 shutdown_connections 2024-01-19T01:15:15.557 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:14.863+0000 7f1b0a565700 1 -- 172.21.15.79:0/1529138212 wait complete. 2024-01-19T01:15:15.558 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-19T01:15:15.558 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-01-19T01:15:15.564 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-19T01:15:15.564 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-01-19T01:15:15.565 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-01-19T01:15:15.565 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-19T01:15:15.566 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph Dashboard is now available at: 2024-01-19T01:15:15.566 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:15.566 INFO:teuthology.orchestra.run.smithi079.stdout: URL: https://smithi079.front.sepia.ceph.com:8443/ 2024-01-19T01:15:15.566 INFO:teuthology.orchestra.run.smithi079.stdout: User: admin 2024-01-19T01:15:15.566 INFO:teuthology.orchestra.run.smithi079.stdout: Password: qs2rohlwhc 2024-01-19T01:15:15.566 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:15.567 INFO:teuthology.orchestra.run.smithi079.stdout:Saving cluster configuration to /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config directory 2024-01-19T01:15:15.567 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling autotune for osd_memory_target 2024-01-19T01:15:16.707 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:16 smithi079 ceph-mon[28374]: mgrmap e13: smithi079.miqczk(active, since 6s) 2024-01-19T01:15:16.710 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.928+0000 7f0636da8700 1 Processor -- start 2024-01-19T01:15:16.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.928+0000 7f0636da8700 1 -- start start 2024-01-19T01:15:16.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.928+0000 7f0636da8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630104ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:16.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.928+0000 7f0636da8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0630105410 con 0x7f0630076c20 2024-01-19T01:15:16.715 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.928+0000 7f0634b44700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630104ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.928+0000 7f0634b44700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630104ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36358/0 (socket says 172.21.15.79:36358) 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.928+0000 7f0634b44700 1 -- 172.21.15.79:0/3247662837 learned_addr learned my addr 172.21.15.79:0/3247662837 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.929+0000 7f0634b44700 1 -- 172.21.15.79:0/3247662837 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630105550 con 0x7f0630076c20 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.929+0000 7f0634b44700 1 --2- 172.21.15.79:0/3247662837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630104ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f06240098e0 tx=0x7f0624009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415fff4b8564b7fb server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.929+0000 7f062f7fe700 1 -- 172.21.15.79:0/3247662837 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f062400b860 con 0x7f0630076c20 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.929+0000 7f062f7fe700 1 -- 172.21.15.79:0/3247662837 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f062400be70 con 0x7f0630076c20 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.929+0000 7f062f7fe700 1 -- 172.21.15.79:0/3247662837 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0624010380 con 0x7f0630076c20 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 -- 172.21.15.79:0/3247662837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 msgr2=0x7f0630104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 --2- 172.21.15.79:0/3247662837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630104ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f06240098e0 tx=0x7f0624009250 comp rx=0 tx=0).stop 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 -- 172.21.15.79:0/3247662837 shutdown_connections 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 --2- 172.21.15.79:0/3247662837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630104ed0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 -- 172.21.15.79:0/3247662837 >> 172.21.15.79:0/3247662837 conn(0x7f06300fc980 msgr2=0x7f06300fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:16.716 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 -- 172.21.15.79:0/3247662837 shutdown_connections 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 -- 172.21.15.79:0/3247662837 wait complete. 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 Processor -- start 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.930+0000 7f0636da8700 1 -- start start 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0636da8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0636da8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0630100dd0 con 0x7f0630076c20 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0634b44700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0634b44700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630102950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36366/0 (socket says 172.21.15.79:36366) 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0634b44700 1 -- 172.21.15.79:0/2143491144 learned_addr learned my addr 172.21.15.79:0/2143491144 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0634b44700 1 -- 172.21.15.79:0/2143491144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240094f0 con 0x7f0630076c20 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0634b44700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630102950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0624006d40 tx=0x7f0624009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f062d7fa700 1 -- 172.21.15.79:0/2143491144 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0624010960 con 0x7f0630076c20 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0630100fd0 con 0x7f0630076c20 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f062d7fa700 1 -- 172.21.15.79:0/2143491144 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f062401a430 con 0x7f0630076c20 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f062d7fa700 1 -- 172.21.15.79:0/2143491144 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06240245b0 con 0x7f0630076c20 2024-01-19T01:15:16.717 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.931+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06301013e0 con 0x7f0630076c20 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.932+0000 7f062d7fa700 1 -- 172.21.15.79:0/2143491144 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f062402d3f0 con 0x7f0630076c20 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.932+0000 7f062d7fa700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06200366e0 0x7f0620038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.933+0000 7f062d7fa700 1 -- 172.21.15.79:0/2143491144 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f062404a3b0 con 0x7f0630076c20 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.933+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063004bb80 con 0x7f0630076c20 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.933+0000 7f062ffff700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06200366e0 0x7f0620038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.933+0000 7f062ffff700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06200366e0 0x7f0620038b90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f061c009990 tx=0x7f061c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:15.937+0000 7f062d7fa700 1 -- 172.21.15.79:0/2143491144 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f062402b8e0 con 0x7f0630076c20 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.125+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f063010aca0 con 0x7f0630076c20 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.125+0000 7f062d7fa700 1 -- 172.21.15.79:0/2143491144 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f062402b2f0 con 0x7f0630076c20 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.127+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06200366e0 msgr2=0x7f0620038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.127+0000 7f0636da8700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06200366e0 0x7f0620038b90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f061c009990 tx=0x7f061c006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.127+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 msgr2=0x7f0630102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.127+0000 7f0636da8700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630102950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0624006d40 tx=0x7f0624009250 comp rx=0 tx=0).stop 2024-01-19T01:15:16.718 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.128+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 shutdown_connections 2024-01-19T01:15:16.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.128+0000 7f0636da8700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06200366e0 0x7f0620038b90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:16.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.128+0000 7f0636da8700 1 --2- 172.21.15.79:0/2143491144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0630076c20 0x7f0630102950 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:16.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.128+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 >> 172.21.15.79:0/2143491144 conn(0x7f06300fc980 msgr2=0x7f06300fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:16.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.128+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 shutdown_connections 2024-01-19T01:15:16.719 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:16.128+0000 7f0636da8700 1 -- 172.21.15.79:0/2143491144 wait complete. 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.160+0000 7f88f968d700 1 Processor -- start 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.160+0000 7f88f968d700 1 -- start start 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.160+0000 7f88f968d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f41026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.160+0000 7f88f968d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f4102c00 con 0x7f88f41022b0 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f41026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f41026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36380/0 (socket says 172.21.15.79:36380) 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f2ffd700 1 -- 172.21.15.79:0/3392481157 learned_addr learned my addr 172.21.15.79:0/3392481157 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f2ffd700 1 -- 172.21.15.79:0/3392481157 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f4102d40 con 0x7f88f41022b0 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f2ffd700 1 --2- 172.21.15.79:0/3392481157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f41026c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098e0 tx=0x7f88e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d909619709a9e9c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:18.038 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f27fc700 1 -- 172.21.15.79:0/3392481157 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88e400b860 con 0x7f88f41022b0 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f27fc700 1 -- 172.21.15.79:0/3392481157 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88e400be70 con 0x7f88f41022b0 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.161+0000 7f88f27fc700 1 -- 172.21.15.79:0/3392481157 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88e4010380 con 0x7f88f41022b0 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 -- 172.21.15.79:0/3392481157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 msgr2=0x7f88f41026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 --2- 172.21.15.79:0/3392481157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f41026c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f88e40098e0 tx=0x7f88e4009250 comp rx=0 tx=0).stop 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 -- 172.21.15.79:0/3392481157 shutdown_connections 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 --2- 172.21.15.79:0/3392481157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f41026c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 -- 172.21.15.79:0/3392481157 >> 172.21.15.79:0/3392481157 conn(0x7f88f40fda10 msgr2=0x7f88f40ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 -- 172.21.15.79:0/3392481157 shutdown_connections 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 -- 172.21.15.79:0/3392481157 wait complete. 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.162+0000 7f88f968d700 1 Processor -- start 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f968d700 1 -- start start 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f968d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f4111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f968d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f41116d0 con 0x7f88f41022b0 2024-01-19T01:15:18.039 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f4111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f2ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f4111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36394/0 (socket says 172.21.15.79:36394) 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f2ffd700 1 -- 172.21.15.79:0/2404131318 learned_addr learned my addr 172.21.15.79:0/2404131318 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f2ffd700 1 -- 172.21.15.79:0/2404131318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e40094f0 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f2ffd700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f4111190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88e4011b60 tx=0x7f88e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88ebfff700 1 -- 172.21.15.79:0/2404131318 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88e4010850 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88ebfff700 1 -- 172.21.15.79:0/2404131318 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88e4010e60 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f41118d0 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88ebfff700 1 -- 172.21.15.79:0/2404131318 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88e401ae20 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.163+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f4111ce0 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.164+0000 7f88ebfff700 1 -- 172.21.15.79:0/2404131318 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f88e40109b0 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.164+0000 7f88ebfff700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f88e0036730 0x7f88e0038be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.164+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f404bb80 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.164+0000 7f88ebfff700 1 -- 172.21.15.79:0/2404131318 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f88e404a450 con 0x7f88f41022b0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.165+0000 7f88ea7fc700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f88e0036730 0x7f88e0038be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:18.040 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.165+0000 7f88ea7fc700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f88e0036730 0x7f88e0038be0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009990 tx=0x7f88dc006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.169+0000 7f88ebfff700 1 -- 172.21.15.79:0/2404131318 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f88e4010c60 con 0x7f88f41022b0 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.456+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f88f4104090 con 0x7f88f41022b0 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.459+0000 7f88ebfff700 1 -- 172.21.15.79:0/2404131318 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f88e4010c60 con 0x7f88f41022b0 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f88e0036730 msgr2=0x7f88e0038be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f88e0036730 0x7f88e0038be0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009990 tx=0x7f88dc006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 msgr2=0x7f88f4111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f4111190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88e4011b60 tx=0x7f88e4009250 comp rx=0 tx=0).stop 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 shutdown_connections 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f88e0036730 0x7f88e0038be0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 --2- 172.21.15.79:0/2404131318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f88f41022b0 0x7f88f4111190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 >> 172.21.15.79:0/2404131318 conn(0x7f88f40fda10 msgr2=0x7f88f40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 shutdown_connections 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr 2024-01-19T01:15:17.461+0000 7f88f968d700 1 -- 172.21.15.79:0/2404131318 wait complete. 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-01-19T01:15:18.041 INFO:teuthology.orchestra.run.smithi079.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout:Or, if you are only running a single cluster on this host: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout:Please consider enabling telemetry to help improve Ceph: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: ceph telemetry on 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout:For more information see: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-01-19T01:15:18.042 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:18.043 INFO:teuthology.orchestra.run.smithi079.stdout:Bootstrap complete. 2024-01-19T01:15:18.074 INFO:tasks.cephadm:Fetching config... 2024-01-19T01:15:18.074 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:15:18.074 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-19T01:15:18.090 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-19T01:15:18.091 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:15:18.091 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-19T01:15:18.147 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-19T01:15:18.147 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:15:18.147 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/keyring of=/dev/stdout 2024-01-19T01:15:18.214 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-19T01:15:18.215 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:15:18.215 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-19T01:15:18.269 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-19T01:15:18.270 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg0aK8WjoCR33NldTWJZDgUoAQgWnfu1uvosYtB1Oy79TNpmCYtIZV53iGu62d0/yZvyxcWAOhtccO6GlOpCk/WHIiDFk6vrST22fP7IqrDBxLHQDvftcggAME3MChuWpK6MKO4+YolIIcXFqnBWBcGgll7lxvkG3leSo+pJKCWePAbpaseAB4skdlBS3z/zl0UDxhZizogTxj1FiEsH+/iKweITTGvq+FZLYzmr6cca2c3fZKLGLCozEANWYyl0hXxgJW5Oykf/HECCt4SV5w8OTgCsaEEyymr+BnU0uGyjcoDj/UQnHKkakV2zEM7jeXmks4vC1/TZYhara690TTuVaEl6+H91bcV7pVJ+nEPfXuUl0OJ+9utOEwKUJtgU+pm4CUItkFsgUEqbtQtoFYyJFCeOvA5qf//T5iEIh8BNNjDROxLRi9BP3frVLla3/IlP2fUUgcWOCnaPE4bCQT/JOmQN0hGMoNA2zns1p4efJUgg3y/OLaWndyfa1WFP0= ceph-f3747e56-b667-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-19T01:15:18.361 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg0aK8WjoCR33NldTWJZDgUoAQgWnfu1uvosYtB1Oy79TNpmCYtIZV53iGu62d0/yZvyxcWAOhtccO6GlOpCk/WHIiDFk6vrST22fP7IqrDBxLHQDvftcggAME3MChuWpK6MKO4+YolIIcXFqnBWBcGgll7lxvkG3leSo+pJKCWePAbpaseAB4skdlBS3z/zl0UDxhZizogTxj1FiEsH+/iKweITTGvq+FZLYzmr6cca2c3fZKLGLCozEANWYyl0hXxgJW5Oykf/HECCt4SV5w8OTgCsaEEyymr+BnU0uGyjcoDj/UQnHKkakV2zEM7jeXmks4vC1/TZYhara690TTuVaEl6+H91bcV7pVJ+nEPfXuUl0OJ+9utOEwKUJtgU+pm4CUItkFsgUEqbtQtoFYyJFCeOvA5qf//T5iEIh8BNNjDROxLRi9BP3frVLla3/IlP2fUUgcWOCnaPE4bCQT/JOmQN0hGMoNA2zns1p4efJUgg3y/OLaWndyfa1WFP0= ceph-f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:15:18.375 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg0aK8WjoCR33NldTWJZDgUoAQgWnfu1uvosYtB1Oy79TNpmCYtIZV53iGu62d0/yZvyxcWAOhtccO6GlOpCk/WHIiDFk6vrST22fP7IqrDBxLHQDvftcggAME3MChuWpK6MKO4+YolIIcXFqnBWBcGgll7lxvkG3leSo+pJKCWePAbpaseAB4skdlBS3z/zl0UDxhZizogTxj1FiEsH+/iKweITTGvq+FZLYzmr6cca2c3fZKLGLCozEANWYyl0hXxgJW5Oykf/HECCt4SV5w8OTgCsaEEyymr+BnU0uGyjcoDj/UQnHKkakV2zEM7jeXmks4vC1/TZYhara690TTuVaEl6+H91bcV7pVJ+nEPfXuUl0OJ+9utOEwKUJtgU+pm4CUItkFsgUEqbtQtoFYyJFCeOvA5qf//T5iEIh8BNNjDROxLRi9BP3frVLla3/IlP2fUUgcWOCnaPE4bCQT/JOmQN0hGMoNA2zns1p4efJUgg3y/OLaWndyfa1WFP0= ceph-f3747e56-b667-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-19T01:15:18.421 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg0aK8WjoCR33NldTWJZDgUoAQgWnfu1uvosYtB1Oy79TNpmCYtIZV53iGu62d0/yZvyxcWAOhtccO6GlOpCk/WHIiDFk6vrST22fP7IqrDBxLHQDvftcggAME3MChuWpK6MKO4+YolIIcXFqnBWBcGgll7lxvkG3leSo+pJKCWePAbpaseAB4skdlBS3z/zl0UDxhZizogTxj1FiEsH+/iKweITTGvq+FZLYzmr6cca2c3fZKLGLCozEANWYyl0hXxgJW5Oykf/HECCt4SV5w8OTgCsaEEyymr+BnU0uGyjcoDj/UQnHKkakV2zEM7jeXmks4vC1/TZYhara690TTuVaEl6+H91bcV7pVJ+nEPfXuUl0OJ+9utOEwKUJtgU+pm4CUItkFsgUEqbtQtoFYyJFCeOvA5qf//T5iEIh8BNNjDROxLRi9BP3frVLla3/IlP2fUUgcWOCnaPE4bCQT/JOmQN0hGMoNA2zns1p4efJUgg3y/OLaWndyfa1WFP0= ceph-f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:15:18.437 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-19T01:15:18.692 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:15:18.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:18 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2404131318' entity='client.admin' 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.839+0000 7fb519ecf700 1 -- 172.21.15.79:0/1578431014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 msgr2=0x7fb5140fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.839+0000 7fb519ecf700 1 --2- 172.21.15.79:0/1578431014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb5140fbb80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5080048c0 tx=0x7fb508009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.839+0000 7fb519ecf700 1 -- 172.21.15.79:0/1578431014 shutdown_connections 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.839+0000 7fb519ecf700 1 --2- 172.21.15.79:0/1578431014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb5140fbb80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.839+0000 7fb519ecf700 1 -- 172.21.15.79:0/1578431014 >> 172.21.15.79:0/1578431014 conn(0x7fb5140f6e50 msgr2=0x7fb5140f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.839+0000 7fb519ecf700 1 -- 172.21.15.79:0/1578431014 shutdown_connections 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.839+0000 7fb519ecf700 1 -- 172.21.15.79:0/1578431014 wait complete. 2024-01-19T01:15:20.842 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb519ecf700 1 Processor -- start 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb519ecf700 1 -- start start 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb519ecf700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb51410f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb519ecf700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5140662c0 con 0x7fb5140fb770 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb5137fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb51410f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb5137fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb51410f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38674/0 (socket says 172.21.15.79:38674) 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb5137fe700 1 -- 172.21.15.79:0/2525800020 learned_addr learned my addr 172.21.15.79:0/2525800020 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.840+0000 7fb5137fe700 1 -- 172.21.15.79:0/2525800020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508005040 con 0x7fb5140fb770 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.841+0000 7fb5137fe700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb51410f780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb508009330 tx=0x7fb508009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.841+0000 7fb4fbfff700 1 -- 172.21.15.79:0/2525800020 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5080096b0 con 0x7fb5140fb770 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.841+0000 7fb4fbfff700 1 -- 172.21.15.79:0/2525800020 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb508009810 con 0x7fb5140fb770 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.841+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5140664c0 con 0x7fb5140fb770 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.841+0000 7fb4fbfff700 1 -- 172.21.15.79:0/2525800020 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb50801c5b0 con 0x7fb5140fb770 2024-01-19T01:15:20.843 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.841+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51410ff00 con 0x7fb5140fb770 2024-01-19T01:15:20.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.842+0000 7fb4fbfff700 1 -- 172.21.15.79:0/2525800020 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fb50801c870 con 0x7fb5140fb770 2024-01-19T01:15:20.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.842+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb514105400 con 0x7fb5140fb770 2024-01-19T01:15:20.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.842+0000 7fb4fbfff700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fb4fc0366d0 0x7fb4fc038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:20.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.842+0000 7fb4fbfff700 1 -- 172.21.15.79:0/2525800020 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb508049a10 con 0x7fb5140fb770 2024-01-19T01:15:20.844 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.842+0000 7fb512ffd700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fb4fc0366d0 0x7fb4fc038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:20.845 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.842+0000 7fb512ffd700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fb4fc0366d0 0x7fb4fc038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb504009990 tx=0x7fb504006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:20.847 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:20.847+0000 7fb4fbfff700 1 -- 172.21.15.79:0/2525800020 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb508011dd0 con 0x7fb5140fb770 2024-01-19T01:15:21.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.033+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fb51404bb80 con 0x7fb5140fb770 2024-01-19T01:15:21.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.036+0000 7fb4fbfff700 1 -- 172.21.15.79:0/2525800020 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb508011370 con 0x7fb5140fb770 2024-01-19T01:15:21.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fb4fc0366d0 msgr2=0x7fb4fc038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:21.039 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fb4fc0366d0 0x7fb4fc038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb504009990 tx=0x7fb504006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 msgr2=0x7fb51410f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb51410f780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb508009330 tx=0x7fb508009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 shutdown_connections 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fb4fc0366d0 0x7fb4fc038b80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 --2- 172.21.15.79:0/2525800020 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb5140fb770 0x7fb51410f780 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 >> 172.21.15.79:0/2525800020 conn(0x7fb5140f6e50 msgr2=0x7fb5140f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 shutdown_connections 2024-01-19T01:15:21.040 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:21.038+0000 7fb519ecf700 1 -- 172.21.15.79:0/2525800020 wait complete. 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: Deploying daemon crash.smithi079 on smithi079 2024-01-19T01:15:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:21 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2525800020' entity='client.admin' 2024-01-19T01:15:21.797 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-19T01:15:21.797 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-19T01:15:22.507 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.772+0000 7f34d5554700 1 -- 172.21.15.79:0/3484727889 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 msgr2=0x7f34d00fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.772+0000 7f34d5554700 1 --2- 172.21.15.79:0/3484727889 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d00fbb90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009c60 tx=0x7f34c8004d10 comp rx=0 tx=0).stop 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.772+0000 7f34d5554700 1 -- 172.21.15.79:0/3484727889 shutdown_connections 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.772+0000 7f34d5554700 1 --2- 172.21.15.79:0/3484727889 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d00fbb90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.772+0000 7f34d5554700 1 -- 172.21.15.79:0/3484727889 >> 172.21.15.79:0/3484727889 conn(0x7f34d00f6e40 msgr2=0x7f34d00f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.772+0000 7f34d5554700 1 -- 172.21.15.79:0/3484727889 shutdown_connections 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.772+0000 7f34d5554700 1 -- 172.21.15.79:0/3484727889 wait complete. 2024-01-19T01:15:24.775 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34d5554700 1 Processor -- start 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34d5554700 1 -- start start 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34d5554700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d010cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34d5554700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d010d0a0 con 0x7f34d00fb780 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34ceffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d010cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34ceffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d010cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38694/0 (socket says 172.21.15.79:38694) 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34ceffd700 1 -- 172.21.15.79:0/612076673 learned_addr learned my addr 172.21.15.79:0/612076673 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.773+0000 7f34ceffd700 1 -- 172.21.15.79:0/612076673 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009940 con 0x7f34d00fb780 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.774+0000 7f34ceffd700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d010cb60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34c8015040 tx=0x7f34c8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.774+0000 7f34b77fe700 1 -- 172.21.15.79:0/612076673 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c800be40 con 0x7f34d00fb780 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.774+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d010d2a0 con 0x7f34d00fb780 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.774+0000 7f34b77fe700 1 -- 172.21.15.79:0/612076673 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34c801d8f0 con 0x7f34d00fb780 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.774+0000 7f34b77fe700 1 -- 172.21.15.79:0/612076673 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c80127e0 con 0x7f34d00fb780 2024-01-19T01:15:24.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.774+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d010df40 con 0x7f34d00fb780 2024-01-19T01:15:24.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.775+0000 7f34b77fe700 1 -- 172.21.15.79:0/612076673 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f34c8019040 con 0x7f34d00fb780 2024-01-19T01:15:24.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.775+0000 7f34b77fe700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f34b80366d0 0x7f34b8038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:24.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.775+0000 7f34b77fe700 1 -- 172.21.15.79:0/612076673 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f34c804a200 con 0x7f34d00fb780 2024-01-19T01:15:24.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.775+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d005f270 con 0x7f34d00fb780 2024-01-19T01:15:24.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.775+0000 7f34ce7fc700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f34b80366d0 0x7f34b8038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:24.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.776+0000 7f34ce7fc700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f34b80366d0 0x7f34b8038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009990 tx=0x7f34c4006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:24.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.780+0000 7f34b77fe700 1 -- 172.21.15.79:0/612076673 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f34c8021080 con 0x7f34d00fb780 2024-01-19T01:15:24.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.988+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f34d0064d80 con 0x7f34b80366d0 2024-01-19T01:15:24.991 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.991+0000 7f34b77fe700 1 -- 172.21.15.79:0/612076673 <== mgr.14158 v2:172.21.15.79:6800/4076006582 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f34d0064d80 con 0x7f34b80366d0 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f34b80366d0 msgr2=0x7f34b8038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f34b80366d0 0x7f34b8038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009990 tx=0x7f34c4006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 msgr2=0x7f34d010cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d010cb60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f34c8015040 tx=0x7f34c8004d10 comp rx=0 tx=0).stop 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 shutdown_connections 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f34b80366d0 0x7f34b8038b80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 --2- 172.21.15.79:0/612076673 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f34d00fb780 0x7f34d010cb60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:24.994 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 >> 172.21.15.79:0/612076673 conn(0x7f34d00f6e40 msgr2=0x7f34d00f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:24.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 shutdown_connections 2024-01-19T01:15:24.995 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:24.993+0000 7f34d5554700 1 -- 172.21.15.79:0/612076673 wait complete. 2024-01-19T01:15:25.686 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-01-19T01:15:25.687 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:15:25.687 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-01-19T01:15:25.706 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:15:25.706 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-19T01:15:25.765 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-01-19T01:15:25.765 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch host add smithi152 2024-01-19T01:15:26.029 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:15:26.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:25 smithi079 ceph-mon[28374]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:15:26.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:25 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:26.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:25 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:26.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:25 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:26.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:25 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:26.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:25 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:27.136 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:26 smithi079 ceph-mon[28374]: Deploying daemon node-exporter.smithi079 on smithi079 2024-01-19T01:15:27.776 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.773+0000 7f067c58c700 1 -- 172.21.15.79:0/1473235912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 msgr2=0x7f06740fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.773+0000 7f067c58c700 1 --2- 172.21.15.79:0/1473235912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f06740fbb80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f06700048c0 tx=0x7f0670009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 -- 172.21.15.79:0/1473235912 shutdown_connections 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 --2- 172.21.15.79:0/1473235912 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f06740fbb80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 -- 172.21.15.79:0/1473235912 >> 172.21.15.79:0/1473235912 conn(0x7f06740f6e50 msgr2=0x7f06740f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 -- 172.21.15.79:0/1473235912 shutdown_connections 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 -- 172.21.15.79:0/1473235912 wait complete. 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 Processor -- start 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 -- start start 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f067410b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067c58c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067410bed0 con 0x7f06740fb770 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067a328700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f067410b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067a328700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f067410b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:38718/0 (socket says 172.21.15.79:38718) 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.774+0000 7f067a328700 1 -- 172.21.15.79:0/3483162188 learned_addr learned my addr 172.21.15.79:0/3483162188 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:27.777 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.775+0000 7f067a328700 1 -- 172.21.15.79:0/3483162188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0670005040 con 0x7f06740fb770 2024-01-19T01:15:27.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.775+0000 7f067a328700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f067410b990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f067000b690 tx=0x7f0670009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:27.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.775+0000 7f066affd700 1 -- 172.21.15.79:0/3483162188 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0670009650 con 0x7f06740fb770 2024-01-19T01:15:27.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.775+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067410c0d0 con 0x7f06740fb770 2024-01-19T01:15:27.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.775+0000 7f066affd700 1 -- 172.21.15.79:0/3483162188 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06700097b0 con 0x7f06740fb770 2024-01-19T01:15:27.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.775+0000 7f066affd700 1 -- 172.21.15.79:0/3483162188 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f067001c5d0 con 0x7f06740fb770 2024-01-19T01:15:27.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.775+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067410cd70 con 0x7f06740fb770 2024-01-19T01:15:27.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.776+0000 7f066affd700 1 -- 172.21.15.79:0/3483162188 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f067001c840 con 0x7f06740fb770 2024-01-19T01:15:27.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.776+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067405f270 con 0x7f06740fb770 2024-01-19T01:15:27.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.776+0000 7f066affd700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06600366d0 0x7f0660038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:27.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.776+0000 7f066affd700 1 -- 172.21.15.79:0/3483162188 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0670049ac0 con 0x7f06740fb770 2024-01-19T01:15:27.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.776+0000 7f0679b27700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06600366d0 0x7f0660038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:27.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.777+0000 7f0679b27700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06600366d0 0x7f0660038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0664009990 tx=0x7f0664006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:27.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.781+0000 7f066affd700 1 -- 172.21.15.79:0/3483162188 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f067004be60 con 0x7f06740fb770 2024-01-19T01:15:27.983 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:27.982+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7f067410d930 con 0x7f06600366d0 2024-01-19T01:15:28.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:28 smithi079 ceph-mon[28374]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:15:30.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.068+0000 7f066affd700 1 -- 172.21.15.79:0/3483162188 <== mgr.14158 v2:172.21.15.79:6800/4076006582 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f067410d930 con 0x7f06600366d0 2024-01-19T01:15:30.070 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-01-19T01:15:30.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06600366d0 msgr2=0x7f0660038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:30.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06600366d0 0x7f0660038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0664009990 tx=0x7f0664006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:30.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 msgr2=0x7f067410b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f067410b990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f067000b690 tx=0x7f0670009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 shutdown_connections 2024-01-19T01:15:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f06600366d0 0x7f0660038b80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 --2- 172.21.15.79:0/3483162188 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f06740fb770 0x7f067410b990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 >> 172.21.15.79:0/3483162188 conn(0x7f06740f6e50 msgr2=0x7f06740f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 shutdown_connections 2024-01-19T01:15:30.073 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:30.071+0000 7f067c58c700 1 -- 172.21.15.79:0/3483162188 wait complete. 2024-01-19T01:15:30.415 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:30 smithi079 ceph-mon[28374]: Deploying cephadm binary to smithi152 2024-01-19T01:15:30.415 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:30 smithi079 ceph-mon[28374]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:30.415 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:30 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:30.415 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:30 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:31.305 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:31 smithi079 ceph-mon[28374]: Added host smithi152 2024-01-19T01:15:31.332 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch host ls --format=json 2024-01-19T01:15:31.794 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:15:32.198 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:32 smithi079 ceph-mon[28374]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:32.198 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:32 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:32.198 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:32 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:32.198 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:32 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:32.198 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:32 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:33.302 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:33 smithi079 ceph-mon[28374]: Deploying daemon alertmanager.smithi079 on smithi079 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.754+0000 7f695d223700 1 -- 172.21.15.79:0/2704844316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 msgr2=0x7f69580fbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.754+0000 7f695d223700 1 --2- 172.21.15.79:0/2704844316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f69580fbb70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f69480048c0 tx=0x7f6948009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.754+0000 7f695d223700 1 -- 172.21.15.79:0/2704844316 shutdown_connections 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.754+0000 7f695d223700 1 --2- 172.21.15.79:0/2704844316 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f69580fbb70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.754+0000 7f695d223700 1 -- 172.21.15.79:0/2704844316 >> 172.21.15.79:0/2704844316 conn(0x7f69580f6ec0 msgr2=0x7f69580f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.755+0000 7f695d223700 1 -- 172.21.15.79:0/2704844316 shutdown_connections 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.755+0000 7f695d223700 1 -- 172.21.15.79:0/2704844316 wait complete. 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.755+0000 7f695d223700 1 Processor -- start 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.755+0000 7f695d223700 1 -- start start 2024-01-19T01:15:33.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f695d223700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f695810b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f695d223700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695810beb0 con 0x7f69580fb760 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f6956d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f695810b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f6956d9d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f695810b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59642/0 (socket says 172.21.15.79:59642) 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f6956d9d700 1 -- 172.21.15.79:0/1742297302 learned_addr learned my addr 172.21.15.79:0/1742297302 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f6956d9d700 1 -- 172.21.15.79:0/1742297302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948005040 con 0x7f69580fb760 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f6956d9d700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f695810b970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f694800b690 tx=0x7f6948009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f693f7fe700 1 -- 172.21.15.79:0/1742297302 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6948009650 con 0x7f69580fb760 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f693f7fe700 1 -- 172.21.15.79:0/1742297302 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69480097b0 con 0x7f69580fb760 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f693f7fe700 1 -- 172.21.15.79:0/1742297302 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f694801c5b0 con 0x7f69580fb760 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695810c0b0 con 0x7f69580fb760 2024-01-19T01:15:33.785 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.756+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695810cd50 con 0x7f69580fb760 2024-01-19T01:15:33.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.757+0000 7f693f7fe700 1 -- 172.21.15.79:0/1742297302 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f6948013420 con 0x7f69580fb760 2024-01-19T01:15:33.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.757+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695805f270 con 0x7f69580fb760 2024-01-19T01:15:33.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.757+0000 7f693f7fe700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f6940036720 0x7f6940038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:33.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.757+0000 7f693f7fe700 1 -- 172.21.15.79:0/1742297302 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f69480498f0 con 0x7f69580fb760 2024-01-19T01:15:33.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.757+0000 7f695659c700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f6940036720 0x7f6940038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:33.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.758+0000 7f695659c700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f6940036720 0x7f6940038bd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f694c009990 tx=0x7f694c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:33.786 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.762+0000 7f693f7fe700 1 -- 172.21.15.79:0/1742297302 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6948011550 con 0x7f69580fb760 2024-01-19T01:15:33.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.967+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f695810d910 con 0x7f6940036720 2024-01-19T01:15:33.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.968+0000 7f693f7fe700 1 -- 172.21.15.79:0/1742297302 <== mgr.14158 v2:172.21.15.79:6800/4076006582 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f695810d910 con 0x7f6940036720 2024-01-19T01:15:33.969 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:15:33.969 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f6940036720 msgr2=0x7f6940038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f6940036720 0x7f6940038bd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f694c009990 tx=0x7f694c006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 msgr2=0x7f695810b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f695810b970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f694800b690 tx=0x7f6948009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 shutdown_connections 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f6940036720 0x7f6940038bd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 --2- 172.21.15.79:0/1742297302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69580fb760 0x7f695810b970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 >> 172.21.15.79:0/1742297302 conn(0x7f69580f6ec0 msgr2=0x7f69580f7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 shutdown_connections 2024-01-19T01:15:33.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:33.970+0000 7f695d223700 1 -- 172.21.15.79:0/1742297302 wait complete. 2024-01-19T01:15:34.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:34 smithi079 ceph-mon[28374]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:34.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:34 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:34.975 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-19T01:15:34.975 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd crush tunables default 2024-01-19T01:15:35.256 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:15:35.595 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:35 smithi079 ceph-mon[28374]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T01:15:36.599 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:36 smithi079 ceph-mon[28374]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 -- 172.21.15.79:0/1033423116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 msgr2=0x7fcec80fb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 --2- 172.21.15.79:0/1033423116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec80fb020 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fceb80048c0 tx=0x7fceb8009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 -- 172.21.15.79:0/1033423116 shutdown_connections 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 --2- 172.21.15.79:0/1033423116 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec80fb020 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 -- 172.21.15.79:0/1033423116 >> 172.21.15.79:0/1033423116 conn(0x7fcec80f65d0 msgr2=0x7fcec80f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 -- 172.21.15.79:0/1033423116 shutdown_connections 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 -- 172.21.15.79:0/1033423116 wait complete. 2024-01-19T01:15:38.090 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 Processor -- start 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.053+0000 7fcecd979700 1 -- start start 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcecd979700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec810ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcecd979700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec810cfd0 con 0x7fcec80fac10 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcec6ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec810ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcec6ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec810ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:59662/0 (socket says 172.21.15.79:59662) 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcec6ffd700 1 -- 172.21.15.79:0/3808282809 learned_addr learned my addr 172.21.15.79:0/3808282809 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcec6ffd700 1 -- 172.21.15.79:0/3808282809 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb8005040 con 0x7fcec80fac10 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcec6ffd700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec810ca90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009330 tx=0x7fceb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fceaf7fe700 1 -- 172.21.15.79:0/3808282809 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fceb80096b0 con 0x7fcec80fac10 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fceaf7fe700 1 -- 172.21.15.79:0/3808282809 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fceb8009810 con 0x7fcec80fac10 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec810d1d0 con 0x7fcec80fac10 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fceaf7fe700 1 -- 172.21.15.79:0/3808282809 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fceb801c5b0 con 0x7fcec80fac10 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.054+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec810de70 con 0x7fcec80fac10 2024-01-19T01:15:38.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.055+0000 7fceaf7fe700 1 -- 172.21.15.79:0/3808282809 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fceb801c870 con 0x7fcec80fac10 2024-01-19T01:15:38.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.055+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec805f270 con 0x7fcec80fac10 2024-01-19T01:15:38.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.055+0000 7fceaf7fe700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fceb00366d0 0x7fceb0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:38.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.055+0000 7fceaf7fe700 1 -- 172.21.15.79:0/3808282809 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fceb80499e0 con 0x7fcec80fac10 2024-01-19T01:15:38.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.055+0000 7fcec67fc700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fceb00366d0 0x7fceb0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:38.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.056+0000 7fcec67fc700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fceb00366d0 0x7fceb0038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009990 tx=0x7fcebc006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:38.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.060+0000 7fceaf7fe700 1 -- 172.21.15.79:0/3808282809 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fceb8025400 con 0x7fcec80fac10 2024-01-19T01:15:38.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.251+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcec810e9c0 con 0x7fcec80fac10 2024-01-19T01:15:38.430 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.429+0000 7fceaf7fe700 1 -- 172.21.15.79:0/3808282809 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fceb8011370 con 0x7fcec80fac10 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fceb00366d0 msgr2=0x7fceb0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fceb00366d0 0x7fceb0038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009990 tx=0x7fcebc006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 msgr2=0x7fcec810ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec810ca90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009330 tx=0x7fceb8009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 shutdown_connections 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fceb00366d0 0x7fceb0038b80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 --2- 172.21.15.79:0/3808282809 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcec80fac10 0x7fcec810ca90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 >> 172.21.15.79:0/3808282809 conn(0x7fcec80f65d0 msgr2=0x7fcec80f7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 shutdown_connections 2024-01-19T01:15:38.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:15:38.432+0000 7fcecd979700 1 -- 172.21.15.79:0/3808282809 wait complete. 2024-01-19T01:15:38.434 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-01-19T01:15:38.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:38 smithi079 ceph-mon[28374]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:38.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:38 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3808282809' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-19T01:15:39.447 INFO:tasks.cephadm:Adding mon.smithi079 on smithi079 2024-01-19T01:15:39.447 INFO:tasks.cephadm:Adding mon.smithi152 on smithi152 2024-01-19T01:15:39.447 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch apply mon '2;smithi079:172.21.15.79=smithi079;smithi152:172.21.15.152=smithi152' 2024-01-19T01:15:39.697 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:39.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:39 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3808282809' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-19T01:15:39.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:39 smithi079 ceph-mon[28374]: osdmap e4: 0 total, 0 up, 0 in 2024-01-19T01:15:39.746 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:40.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:40 smithi079 ceph-mon[28374]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:41.268 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 -- 172.21.15.152:0/1342788691 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 msgr2=0x7fdea40fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:41.268 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 --2- 172.21.15.152:0/1342788691 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea40fcc90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdea00048c0 tx=0x7fdea0009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 -- 172.21.15.152:0/1342788691 shutdown_connections 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 --2- 172.21.15.152:0/1342788691 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea40fcc90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 -- 172.21.15.152:0/1342788691 >> 172.21.15.152:0/1342788691 conn(0x7fdea40f7f40 msgr2=0x7fdea40fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 -- 172.21.15.152:0/1342788691 shutdown_connections 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 -- 172.21.15.152:0/1342788691 wait complete. 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 Processor -- start 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.265+0000 7fdeab712700 1 -- start start 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdeab712700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea4071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdeab712700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea406ff90 con 0x7fdea40fc880 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdea94ae700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea4071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdea94ae700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea4071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:41366/0 (socket says 172.21.15.152:41366) 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdea94ae700 1 -- 172.21.15.152:0/2606325013 learned_addr learned my addr 172.21.15.152:0/2606325013 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:15:41.269 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdea94ae700 1 -- 172.21.15.152:0/2606325013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea0005040 con 0x7fdea40fc880 2024-01-19T01:15:41.270 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdea94ae700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea4071b10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009330 tx=0x7fdea0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:41.270 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fde99ffb700 1 -- 172.21.15.152:0/2606325013 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdea00096b0 con 0x7fdea40fc880 2024-01-19T01:15:41.270 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.266+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea4070190 con 0x7fdea40fc880 2024-01-19T01:15:41.270 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.267+0000 7fde99ffb700 1 -- 172.21.15.152:0/2606325013 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdea0009810 con 0x7fdea40fc880 2024-01-19T01:15:41.270 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.267+0000 7fde99ffb700 1 -- 172.21.15.152:0/2606325013 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdea001c5b0 con 0x7fdea40fc880 2024-01-19T01:15:41.270 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.267+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea4070680 con 0x7fdea40fc880 2024-01-19T01:15:41.270 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.268+0000 7fde99ffb700 1 -- 172.21.15.152:0/2606325013 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fdea0013420 con 0x7fdea40fc880 2024-01-19T01:15:41.271 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.268+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea4106510 con 0x7fdea40fc880 2024-01-19T01:15:41.271 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.268+0000 7fde99ffb700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fde900366d0 0x7fde90038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:41.271 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.268+0000 7fde99ffb700 1 -- 172.21.15.152:0/2606325013 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdea0049ac0 con 0x7fdea40fc880 2024-01-19T01:15:41.271 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.268+0000 7fdea8cad700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fde900366d0 0x7fde90038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:41.271 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.269+0000 7fdea8cad700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fde900366d0 0x7fde90038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fde94009990 tx=0x7fde94006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:41.274 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.273+0000 7fde99ffb700 1 -- 172.21.15.152:0/2606325013 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdea00119c0 con 0x7fdea40fc880 2024-01-19T01:15:41.467 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.466+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 --> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi152:172.21.15.152=smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7fdea402b300 con 0x7fde900366d0 2024-01-19T01:15:41.473 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.473+0000 7fde99ffb700 1 -- 172.21.15.152:0/2606325013 <== mgr.14158 v2:172.21.15.79:6800/4076006582 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdea402b300 con 0x7fde900366d0 2024-01-19T01:15:41.474 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mon update... 2024-01-19T01:15:41.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fde900366d0 msgr2=0x7fde90038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fde900366d0 0x7fde90038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fde94009990 tx=0x7fde94006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 msgr2=0x7fdea4071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea4071b10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009330 tx=0x7fdea0009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 shutdown_connections 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fde900366d0 0x7fde90038b80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 --2- 172.21.15.152:0/2606325013 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdea40fc880 0x7fdea4071b10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.474+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 >> 172.21.15.152:0/2606325013 conn(0x7fdea40f7f40 msgr2=0x7fdea40f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.475+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 shutdown_connections 2024-01-19T01:15:41.476 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:41.475+0000 7fdeab712700 1 -- 172.21.15.152:0/2606325013 wait complete. 2024-01-19T01:15:42.036 DEBUG:teuthology.orchestra.run.smithi152:mon.smithi152> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@mon.smithi152.service 2024-01-19T01:15:42.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:15:42.039 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:15:42.316 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:42.364 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:42.496 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:42 smithi079 ceph-mon[28374]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:42.496 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:42 smithi079 ceph-mon[28374]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi152:172.21.15.152=smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:15:42.496 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:42 smithi079 ceph-mon[28374]: Saving service mon spec with placement smithi079:172.21.15.79=smithi079;smithi152:172.21.15.152=smithi152;count:2 2024-01-19T01:15:42.496 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:42 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:43.725 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.724+0000 7f4b139b1700 1 -- 172.21.15.152:0/3499298421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 msgr2=0x7f4b0c0fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:43.726 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.724+0000 7f4b139b1700 1 --2- 172.21.15.152:0/3499298421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c0fbb80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0048c0 tx=0x7f4afc009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:43.726 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.724+0000 7f4b139b1700 1 -- 172.21.15.152:0/3499298421 shutdown_connections 2024-01-19T01:15:43.726 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.724+0000 7f4b139b1700 1 --2- 172.21.15.152:0/3499298421 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c0fbb80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:43.726 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.724+0000 7f4b139b1700 1 -- 172.21.15.152:0/3499298421 >> 172.21.15.152:0/3499298421 conn(0x7f4b0c0f6e50 msgr2=0x7f4b0c0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:43.726 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.724+0000 7f4b139b1700 1 -- 172.21.15.152:0/3499298421 shutdown_connections 2024-01-19T01:15:43.726 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.724+0000 7f4b139b1700 1 -- 172.21.15.152:0/3499298421 wait complete. 2024-01-19T01:15:43.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.725+0000 7f4b139b1700 1 Processor -- start 2024-01-19T01:15:43.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.725+0000 7f4b139b1700 1 -- start start 2024-01-19T01:15:43.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.725+0000 7f4b139b1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c10ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:43.727 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.725+0000 7f4b139b1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c10cfd0 con 0x7f4b0c0fb770 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.725+0000 7f4b1174d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c10ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.725+0000 7f4b1174d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c10ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:41382/0 (socket says 172.21.15.152:41382) 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.725+0000 7f4b1174d700 1 -- 172.21.15.152:0/1217706497 learned_addr learned my addr 172.21.15.152:0/1217706497 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.726+0000 7f4b1174d700 1 -- 172.21.15.152:0/1217706497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc005040 con 0x7f4b0c0fb770 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.726+0000 7f4b1174d700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c10ca90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009330 tx=0x7f4afc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.726+0000 7f4b01ffb700 1 -- 172.21.15.152:0/1217706497 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4afc0096b0 con 0x7f4b0c0fb770 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.726+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c10d1d0 con 0x7f4b0c0fb770 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.726+0000 7f4b01ffb700 1 -- 172.21.15.152:0/1217706497 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4afc009810 con 0x7f4b0c0fb770 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.726+0000 7f4b01ffb700 1 -- 172.21.15.152:0/1217706497 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4afc01c5b0 con 0x7f4b0c0fb770 2024-01-19T01:15:43.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.726+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c10de70 con 0x7f4b0c0fb770 2024-01-19T01:15:43.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.727+0000 7f4b01ffb700 1 -- 172.21.15.152:0/1217706497 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f4afc01c870 con 0x7f4b0c0fb770 2024-01-19T01:15:43.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.727+0000 7f4b01ffb700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f4af80366d0 0x7f4af8038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:43.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.727+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c05f270 con 0x7f4b0c0fb770 2024-01-19T01:15:43.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.727+0000 7f4b01ffb700 1 -- 172.21.15.152:0/1217706497 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4afc049cd0 con 0x7f4b0c0fb770 2024-01-19T01:15:43.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.728+0000 7f4b10f4c700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f4af80366d0 0x7f4af8038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:43.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.728+0000 7f4b10f4c700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f4af80366d0 0x7f4af8038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009990 tx=0x7f4b08006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:43.733 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.733+0000 7f4b01ffb700 1 -- 172.21.15.152:0/1217706497 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4afc011dd0 con 0x7f4b0c0fb770 2024-01-19T01:15:43.988 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.987+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b0c10e1f0 con 0x7f4b0c0fb770 2024-01-19T01:15:43.989 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.988+0000 7f4b01ffb700 1 -- 172.21.15.152:0/1217706497 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4afc011370 con 0x7f4b0c0fb770 2024-01-19T01:15:43.989 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:15:43.989 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f4af80366d0 msgr2=0x7f4af8038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f4af80366d0 0x7f4af8038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009990 tx=0x7f4b08006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 msgr2=0x7f4b0c10ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c10ca90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009330 tx=0x7f4afc009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 shutdown_connections 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f4af80366d0 0x7f4af8038b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 --2- 172.21.15.152:0/1217706497 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4b0c0fb770 0x7f4b0c10ca90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:43.991 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 >> 172.21.15.152:0/1217706497 conn(0x7f4b0c0f6e50 msgr2=0x7f4b0c0f7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:43.992 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 shutdown_connections 2024-01-19T01:15:43.992 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:43.990+0000 7f4b139b1700 1 -- 172.21.15.152:0/1217706497 wait complete. 2024-01-19T01:15:43.992 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:15:44.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:44.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: Deploying daemon grafana.smithi079 on smithi079 2024-01-19T01:15:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:43 smithi079 ceph-mon[28374]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:44.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:44 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/1217706497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:15:44.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:44 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:45.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:15:45.482 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:15:45.731 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:45.779 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:45.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:45 smithi079 ceph-mon[28374]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.308+0000 7f9386e16700 1 -- 172.21.15.152:0/816955700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 msgr2=0x7f93800ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.308+0000 7f9386e16700 1 --2- 172.21.15.152:0/816955700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f93800ff450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f93740048c0 tx=0x7f9374009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 -- 172.21.15.152:0/816955700 shutdown_connections 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 --2- 172.21.15.152:0/816955700 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f93800ff450 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 -- 172.21.15.152:0/816955700 >> 172.21.15.152:0/816955700 conn(0x7f93800f6ec0 msgr2=0x7f93800f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 -- 172.21.15.152:0/816955700 shutdown_connections 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 -- 172.21.15.152:0/816955700 wait complete. 2024-01-19T01:15:47.313 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 Processor -- start 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 -- start start 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f938010cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9386e16700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938010d000 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.309+0000 7f9384bb2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f938010cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f9384bb2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f938010cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:41398/0 (socket says 172.21.15.152:41398) 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f9384bb2700 1 -- 172.21.15.152:0/728199680 learned_addr learned my addr 172.21.15.152:0/728199680 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f9384bb2700 1 -- 172.21.15.152:0/728199680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9374005040 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f9384bb2700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f938010cac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9374009330 tx=0x7f9374009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f937d7fa700 1 -- 172.21.15.152:0/728199680 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93740096b0 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938010d200 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f937d7fa700 1 -- 172.21.15.152:0/728199680 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9374009810 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f937d7fa700 1 -- 172.21.15.152:0/728199680 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f937401c5b0 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.310+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938010dea0 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.311+0000 7f937d7fa700 1 -- 172.21.15.152:0/728199680 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f937401c870 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.311+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938005f270 con 0x7f9380066530 2024-01-19T01:15:47.314 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.311+0000 7f937d7fa700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f93700366d0 0x7f9370038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:47.315 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.311+0000 7f937d7fa700 1 -- 172.21.15.152:0/728199680 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9374049bc0 con 0x7f9380066530 2024-01-19T01:15:47.315 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.311+0000 7f937ffff700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f93700366d0 0x7f9370038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:47.315 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.312+0000 7f937ffff700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f93700366d0 0x7f9370038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f936c009990 tx=0x7f936c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:47.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.316+0000 7f937d7fa700 1 -- 172.21.15.152:0/728199680 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9374011550 con 0x7f9380066530 2024-01-19T01:15:47.575 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.574+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f938010e9f0 con 0x7f9380066530 2024-01-19T01:15:47.576 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.575+0000 7f937d7fa700 1 -- 172.21.15.152:0/728199680 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9374011370 con 0x7f9380066530 2024-01-19T01:15:47.576 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:15:47.576 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f93700366d0 msgr2=0x7f9370038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f93700366d0 0x7f9370038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f936c009990 tx=0x7f936c006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 msgr2=0x7f938010cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f938010cac0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9374009330 tx=0x7f9374009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 shutdown_connections 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f93700366d0 0x7f9370038b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 --2- 172.21.15.152:0/728199680 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9380066530 0x7f938010cac0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 >> 172.21.15.152:0/728199680 conn(0x7f93800f6ec0 msgr2=0x7f93800f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 shutdown_connections 2024-01-19T01:15:47.578 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:47.577+0000 7f9386e16700 1 -- 172.21.15.152:0/728199680 wait complete. 2024-01-19T01:15:47.579 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:15:48.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:48 smithi079 ceph-mon[28374]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:48.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:48 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/728199680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:15:49.210 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:15:49.210 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:15:49.461 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:49.515 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:50.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:50 smithi079 ceph-mon[28374]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:50.872 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.870+0000 7fd331d68700 1 -- 172.21.15.152:0/3490083057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 msgr2=0x7fd32c0ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:50.872 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.870+0000 7fd331d68700 1 --2- 172.21.15.152:0/3490083057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c0ff7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0048c0 tx=0x7fd31c009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:50.872 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.870+0000 7fd331d68700 1 -- 172.21.15.152:0/3490083057 shutdown_connections 2024-01-19T01:15:50.872 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.870+0000 7fd331d68700 1 --2- 172.21.15.152:0/3490083057 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c0ff7a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:50.872 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.870+0000 7fd331d68700 1 -- 172.21.15.152:0/3490083057 >> 172.21.15.152:0/3490083057 conn(0x7fd32c0f6e40 msgr2=0x7fd32c0f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.870+0000 7fd331d68700 1 -- 172.21.15.152:0/3490083057 shutdown_connections 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.870+0000 7fd331d68700 1 -- 172.21.15.152:0/3490083057 wait complete. 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.871+0000 7fd331d68700 1 Processor -- start 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.871+0000 7fd331d68700 1 -- start start 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.871+0000 7fd331d68700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c10ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.871+0000 7fd331d68700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c10cfa0 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.871+0000 7fd32b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c10ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.871+0000 7fd32b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c10ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:38316/0 (socket says 172.21.15.152:38316) 2024-01-19T01:15:50.873 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.871+0000 7fd32b7fe700 1 -- 172.21.15.152:0/3087720042 learned_addr learned my addr 172.21.15.152:0/3087720042 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:15:50.874 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.872+0000 7fd32b7fe700 1 -- 172.21.15.152:0/3087720042 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c005040 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.872+0000 7fd32b7fe700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c10ca60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00b690 tx=0x7fd31c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.872+0000 7fd328ff9700 1 -- 172.21.15.152:0/3087720042 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd31c009650 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.872+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c10d1a0 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.872+0000 7fd328ff9700 1 -- 172.21.15.152:0/3087720042 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd31c0097b0 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.872+0000 7fd328ff9700 1 -- 172.21.15.152:0/3087720042 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd31c01c5b0 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.872+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c10de40 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.873+0000 7fd328ff9700 1 -- 172.21.15.152:0/3087720042 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fd31c013420 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.873+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c05f270 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.873+0000 7fd328ff9700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd318036720 0x7fd318038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.873+0000 7fd328ff9700 1 -- 172.21.15.152:0/3087720042 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd31c0498f0 con 0x7fd32c0fd3c0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.873+0000 7fd322ffd700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd318036720 0x7fd318038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:50.875 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.874+0000 7fd322ffd700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd318036720 0x7fd318038bd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd314009990 tx=0x7fd314006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:50.879 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:50.878+0000 7fd328ff9700 1 -- 172.21.15.152:0/3087720042 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd31c011550 con 0x7fd32c0fd3c0 2024-01-19T01:15:51.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.136+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd32c10e990 con 0x7fd32c0fd3c0 2024-01-19T01:15:51.137 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.136+0000 7fd328ff9700 1 -- 172.21.15.152:0/3087720042 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd31c011370 con 0x7fd32c0fd3c0 2024-01-19T01:15:51.137 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:15:51.138 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd318036720 msgr2=0x7fd318038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd318036720 0x7fd318038bd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd314009990 tx=0x7fd314006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 msgr2=0x7fd32c10ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c10ca60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00b690 tx=0x7fd31c009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 shutdown_connections 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd318036720 0x7fd318038bd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 --2- 172.21.15.152:0/3087720042 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd32c0fd3c0 0x7fd32c10ca60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.138+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 >> 172.21.15.152:0/3087720042 conn(0x7fd32c0f6e40 msgr2=0x7fd32c0f7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.139+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 shutdown_connections 2024-01-19T01:15:51.140 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:51.139+0000 7fd331d68700 1 -- 172.21.15.152:0/3087720042 wait complete. 2024-01-19T01:15:51.141 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:15:51.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:51 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/3087720042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:15:52.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:52 smithi079 ceph-mon[28374]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:52.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:15:52.813 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:15:53.061 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:53.107 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.488+0000 7fd6384a3700 1 -- 172.21.15.152:0/492190620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 msgr2=0x7fd6300fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.488+0000 7fd6384a3700 1 --2- 172.21.15.152:0/492190620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd6300fcc80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009c60 tx=0x7fd62c004d10 comp rx=0 tx=0).stop 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 -- 172.21.15.152:0/492190620 shutdown_connections 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 --2- 172.21.15.152:0/492190620 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd6300fcc80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 -- 172.21.15.152:0/492190620 >> 172.21.15.152:0/492190620 conn(0x7fd6300f7f50 msgr2=0x7fd6300fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 -- 172.21.15.152:0/492190620 shutdown_connections 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 -- 172.21.15.152:0/492190620 wait complete. 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 Processor -- start 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 -- start start 2024-01-19T01:15:54.491 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd63010cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:54.492 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.489+0000 7fd6384a3700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63010d050 con 0x7fd6300fc870 2024-01-19T01:15:54.492 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd63623f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd63010cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:54.492 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd63623f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd63010cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:38338/0 (socket says 172.21.15.152:38338) 2024-01-19T01:15:54.492 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd63623f700 1 -- 172.21.15.152:0/4233997397 learned_addr learned my addr 172.21.15.152:0/4233997397 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:15:54.492 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd63623f700 1 -- 172.21.15.152:0/4233997397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c009940 con 0x7fd6300fc870 2024-01-19T01:15:54.492 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd63623f700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd63010cb10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd62c015040 tx=0x7fd62c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:54.493 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd626ffd700 1 -- 172.21.15.152:0/4233997397 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd62c00be40 con 0x7fd6300fc870 2024-01-19T01:15:54.493 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63010d250 con 0x7fd6300fc870 2024-01-19T01:15:54.493 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd626ffd700 1 -- 172.21.15.152:0/4233997397 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd62c01d8f0 con 0x7fd6300fc870 2024-01-19T01:15:54.493 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd626ffd700 1 -- 172.21.15.152:0/4233997397 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd62c0127e0 con 0x7fd6300fc870 2024-01-19T01:15:54.493 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.490+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63010def0 con 0x7fd6300fc870 2024-01-19T01:15:54.494 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.491+0000 7fd626ffd700 1 -- 172.21.15.152:0/4233997397 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fd62c019040 con 0x7fd6300fc870 2024-01-19T01:15:54.494 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.491+0000 7fd626ffd700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd61c036720 0x7fd61c038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:54.494 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.491+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63005f270 con 0x7fd6300fc870 2024-01-19T01:15:54.494 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.491+0000 7fd626ffd700 1 -- 172.21.15.152:0/4233997397 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd62c04a2b0 con 0x7fd6300fc870 2024-01-19T01:15:54.494 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.492+0000 7fd635a3e700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd61c036720 0x7fd61c038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:54.494 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.492+0000 7fd635a3e700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd61c036720 0x7fd61c038bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd620009990 tx=0x7fd620006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:54.497 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.497+0000 7fd626ffd700 1 -- 172.21.15.152:0/4233997397 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd62c017070 con 0x7fd6300fc870 2024-01-19T01:15:54.726 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:54 smithi079 ceph-mon[28374]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:54.753 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.752+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd63010e9b0 con 0x7fd6300fc870 2024-01-19T01:15:54.754 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.753+0000 7fd626ffd700 1 -- 172.21.15.152:0/4233997397 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd62c01d620 con 0x7fd6300fc870 2024-01-19T01:15:54.754 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:15:54.754 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:15:54.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd61c036720 msgr2=0x7fd61c038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:54.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd61c036720 0x7fd61c038bd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd620009990 tx=0x7fd620006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:54.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 msgr2=0x7fd63010cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:54.756 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd63010cb10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd62c015040 tx=0x7fd62c004d10 comp rx=0 tx=0).stop 2024-01-19T01:15:54.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 shutdown_connections 2024-01-19T01:15:54.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fd61c036720 0x7fd61c038bd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:54.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 --2- 172.21.15.152:0/4233997397 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fd6300fc870 0x7fd63010cb10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:54.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 >> 172.21.15.152:0/4233997397 conn(0x7fd6300f7f50 msgr2=0x7fd6300f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:54.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 shutdown_connections 2024-01-19T01:15:54.757 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:54.755+0000 7fd6384a3700 1 -- 172.21.15.152:0/4233997397 wait complete. 2024-01-19T01:15:54.757 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:15:55.721 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:55 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/4233997397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:15:56.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:15:56.350 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:15:56.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:56 smithi079 ceph-mon[28374]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:56.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:56 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:56.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:56 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:56.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:56 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:56.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:56 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:56.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:56 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:56.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:56 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:15:56.599 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:56.645 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:15:57.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:57 smithi079 ceph-mon[28374]: Deploying daemon prometheus.smithi079 on smithi079 2024-01-19T01:15:58.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 -- 172.21.15.152:0/3729088076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 msgr2=0x7f57c40ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:58.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 --2- 172.21.15.152:0/3729088076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c40ff440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f57c00048c0 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:58.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 -- 172.21.15.152:0/3729088076 shutdown_connections 2024-01-19T01:15:58.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 --2- 172.21.15.152:0/3729088076 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c40ff440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:58.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 -- 172.21.15.152:0/3729088076 >> 172.21.15.152:0/3729088076 conn(0x7f57c40f6e50 msgr2=0x7f57c40f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:58.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 -- 172.21.15.152:0/3729088076 shutdown_connections 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 -- 172.21.15.152:0/3729088076 wait complete. 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 Processor -- start 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.015+0000 7f57cb485700 1 -- start start 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57cb485700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c410ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57cb485700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c410cf90 con 0x7f57c4066530 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57c9221700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c410ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57c9221700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c410ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:40980/0 (socket says 172.21.15.152:40980) 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57c9221700 1 -- 172.21.15.152:0/2547156959 learned_addr learned my addr 172.21.15.152:0/2547156959 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57c9221700 1 -- 172.21.15.152:0/2547156959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0005040 con 0x7f57c4066530 2024-01-19T01:15:58.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57c9221700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c410ca50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f57c000b690 tx=0x7f57c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:58.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57b9ffb700 1 -- 172.21.15.152:0/2547156959 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c0009650 con 0x7f57c4066530 2024-01-19T01:15:58.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c410d190 con 0x7f57c4066530 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57b9ffb700 1 -- 172.21.15.152:0/2547156959 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57c00097b0 con 0x7f57c4066530 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.016+0000 7f57b9ffb700 1 -- 172.21.15.152:0/2547156959 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c001c5b0 con 0x7f57c4066530 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.017+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c410de30 con 0x7f57c4066530 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.017+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c405f270 con 0x7f57c4066530 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.018+0000 7f57b9ffb700 1 -- 172.21.15.152:0/2547156959 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f57c0013a60 con 0x7f57c4066530 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.018+0000 7f57b9ffb700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f57b00366d0 0x7f57b0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.018+0000 7f57b9ffb700 1 -- 172.21.15.152:0/2547156959 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f57c0049910 con 0x7f57c4066530 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.018+0000 7f57c8a20700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f57b00366d0 0x7f57b0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:15:58.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.018+0000 7f57c8a20700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f57b00366d0 0x7f57b0038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009990 tx=0x7f57b4006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:15:58.023 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.022+0000 7f57b9ffb700 1 -- 172.21.15.152:0/2547156959 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f57c0011550 con 0x7f57c4066530 2024-01-19T01:15:58.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.279+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57c410e980 con 0x7f57c4066530 2024-01-19T01:15:58.280 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.279+0000 7f57b9ffb700 1 -- 172.21.15.152:0/2547156959 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f57c0011370 con 0x7f57c4066530 2024-01-19T01:15:58.281 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:15:58.281 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.281+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f57b00366d0 msgr2=0x7f57b0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.281+0000 7f57cb485700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f57b00366d0 0x7f57b0038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009990 tx=0x7f57b4006e40 comp rx=0 tx=0).stop 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.281+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 msgr2=0x7f57c410ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.281+0000 7f57cb485700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c410ca50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f57c000b690 tx=0x7f57c0009e30 comp rx=0 tx=0).stop 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.281+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 shutdown_connections 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.282+0000 7f57cb485700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f57b00366d0 0x7f57b0038b80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.282+0000 7f57cb485700 1 --2- 172.21.15.152:0/2547156959 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f57c4066530 0x7f57c410ca50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.282+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 >> 172.21.15.152:0/2547156959 conn(0x7f57c40f6e50 msgr2=0x7f57c40f7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.282+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 shutdown_connections 2024-01-19T01:15:58.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:15:58.282+0000 7f57cb485700 1 -- 172.21.15.152:0/2547156959 wait complete. 2024-01-19T01:15:58.284 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:15:58.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:58 smithi079 ceph-mon[28374]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:15:58.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:15:58 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/2547156959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:15:59.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:15:59.855 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:00.111 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:00.157 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:00.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:00 smithi079 ceph-mon[28374]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:00.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:00 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:01.659 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.657+0000 7f802e1e2700 1 -- 172.21.15.152:0/2226801764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 msgr2=0x7f80280fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:01.659 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.657+0000 7f802e1e2700 1 --2- 172.21.15.152:0/2226801764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f80280fbb90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f80180048c0 tx=0x7f8018009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:01.659 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.657+0000 7f802e1e2700 1 -- 172.21.15.152:0/2226801764 shutdown_connections 2024-01-19T01:16:01.659 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.657+0000 7f802e1e2700 1 --2- 172.21.15.152:0/2226801764 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f80280fbb90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:01.659 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.657+0000 7f802e1e2700 1 -- 172.21.15.152:0/2226801764 >> 172.21.15.152:0/2226801764 conn(0x7f80280f6e40 msgr2=0x7f80280f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:01.659 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.657+0000 7f802e1e2700 1 -- 172.21.15.152:0/2226801764 shutdown_connections 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.657+0000 7f802e1e2700 1 -- 172.21.15.152:0/2226801764 wait complete. 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.658+0000 7f802e1e2700 1 Processor -- start 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.658+0000 7f802e1e2700 1 -- start start 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.658+0000 7f802e1e2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f8028111bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.658+0000 7f802e1e2700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802810d5b0 con 0x7f80280fb780 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.658+0000 7f80277fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f8028111bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.658+0000 7f80277fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f8028111bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:41008/0 (socket says 172.21.15.152:41008) 2024-01-19T01:16:01.660 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.658+0000 7f80277fe700 1 -- 172.21.15.152:0/1660464073 learned_addr learned my addr 172.21.15.152:0/1660464073 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.659+0000 7f80277fe700 1 -- 172.21.15.152:0/1660464073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8018005040 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.659+0000 7f80277fe700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f8028111bc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8018009330 tx=0x7f8018009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.659+0000 7f8024ff9700 1 -- 172.21.15.152:0/1660464073 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80180096b0 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.659+0000 7f8024ff9700 1 -- 172.21.15.152:0/1660464073 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8018009810 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.659+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8028065a10 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.659+0000 7f8024ff9700 1 -- 172.21.15.152:0/1660464073 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f801801c5b0 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.659+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8028112100 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.660+0000 7f8024ff9700 1 -- 172.21.15.152:0/1660464073 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f801801c870 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.660+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8028105410 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.660+0000 7f8024ff9700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f80140366d0 0x7f8014038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.660+0000 7f8024ff9700 1 -- 172.21.15.152:0/1660464073 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8018049b90 con 0x7f80280fb780 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.660+0000 7f801effd700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f80140366d0 0x7f8014038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:01.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.661+0000 7f801effd700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f80140366d0 0x7f8014038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8010009990 tx=0x7f8010006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:01.667 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.665+0000 7f8024ff9700 1 -- 172.21.15.152:0/1660464073 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8018011370 con 0x7f80280fb780 2024-01-19T01:16:01.921 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.920+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8028105410 con 0x7f80280fb780 2024-01-19T01:16:01.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.921+0000 7f8024ff9700 1 -- 172.21.15.152:0/1660464073 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8018011370 con 0x7f80280fb780 2024-01-19T01:16:01.922 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:01.922 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:01.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f80140366d0 msgr2=0x7f8014038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:01.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f80140366d0 0x7f8014038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8010009990 tx=0x7f8010006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:01.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 msgr2=0x7f8028111bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:01.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f8028111bc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8018009330 tx=0x7f8018009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:01.924 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 shutdown_connections 2024-01-19T01:16:01.925 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f80140366d0 0x7f8014038b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:01.925 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 --2- 172.21.15.152:0/1660464073 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f80280fb780 0x7f8028111bc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:01.925 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 >> 172.21.15.152:0/1660464073 conn(0x7f80280f6e40 msgr2=0x7f80280f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:01.925 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 shutdown_connections 2024-01-19T01:16:01.925 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:01.923+0000 7f802e1e2700 1 -- 172.21.15.152:0/1660464073 wait complete. 2024-01-19T01:16:01.925 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:02.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:02 smithi079 ceph-mon[28374]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:02.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:02 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/1660464073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:03.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:03.458 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:03.707 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:03.754 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:04.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:04 smithi079 ceph-mon[28374]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:05.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.126+0000 7f94c8cf6700 1 -- 172.21.15.152:0/4225582572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 msgr2=0x7f94c40fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:05.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.126+0000 7f94c8cf6700 1 --2- 172.21.15.152:0/4225582572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c40fcc80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009c60 tx=0x7f94bc004d10 comp rx=0 tx=0).stop 2024-01-19T01:16:05.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.126+0000 7f94c8cf6700 1 -- 172.21.15.152:0/4225582572 shutdown_connections 2024-01-19T01:16:05.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.126+0000 7f94c8cf6700 1 --2- 172.21.15.152:0/4225582572 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c40fcc80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:05.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.126+0000 7f94c8cf6700 1 -- 172.21.15.152:0/4225582572 >> 172.21.15.152:0/4225582572 conn(0x7f94c40f7f50 msgr2=0x7f94c40fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:05.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.126+0000 7f94c8cf6700 1 -- 172.21.15.152:0/4225582572 shutdown_connections 2024-01-19T01:16:05.128 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.126+0000 7f94c8cf6700 1 -- 172.21.15.152:0/4225582572 wait complete. 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c8cf6700 1 Processor -- start 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c8cf6700 1 -- start start 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c8cf6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c410cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c8cf6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c410d0e0 con 0x7f94c40fc870 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c410cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c259c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c410cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:41020/0 (socket says 172.21.15.152:41020) 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c259c700 1 -- 172.21.15.152:0/3769598103 learned_addr learned my addr 172.21.15.152:0/3769598103 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:05.129 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.127+0000 7f94c259c700 1 -- 172.21.15.152:0/3769598103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc009940 con 0x7f94c40fc870 2024-01-19T01:16:05.130 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.128+0000 7f94c259c700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c410cba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f94bc000f80 tx=0x7f94bc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:05.130 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.128+0000 7f94b2ffd700 1 -- 172.21.15.152:0/3769598103 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94bc00bec0 con 0x7f94c40fc870 2024-01-19T01:16:05.130 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.128+0000 7f94b2ffd700 1 -- 172.21.15.152:0/3769598103 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94bc01d8f0 con 0x7f94c40fc870 2024-01-19T01:16:05.130 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.128+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c410d2e0 con 0x7f94c40fc870 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.128+0000 7f94b2ffd700 1 -- 172.21.15.152:0/3769598103 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94bc012890 con 0x7f94c40fc870 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.128+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c410df80 con 0x7f94c40fc870 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.129+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c405f270 con 0x7f94c40fc870 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.129+0000 7f94b2ffd700 1 -- 172.21.15.152:0/3769598103 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f94bc019040 con 0x7f94c40fc870 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.129+0000 7f94b2ffd700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f94ac0366d0 0x7f94ac038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.129+0000 7f94b2ffd700 1 -- 172.21.15.152:0/3769598103 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f94bc04a1f0 con 0x7f94c40fc870 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.129+0000 7f94c1d9b700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f94ac0366d0 0x7f94ac038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:05.131 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.129+0000 7f94c1d9b700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f94ac0366d0 0x7f94ac038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009990 tx=0x7f94b8006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:05.135 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.134+0000 7f94b2ffd700 1 -- 172.21.15.152:0/3769598103 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f94bc02e350 con 0x7f94c40fc870 2024-01-19T01:16:05.389 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.388+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94c410ea70 con 0x7f94c40fc870 2024-01-19T01:16:05.390 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.389+0000 7f94b2ffd700 1 -- 172.21.15.152:0/3769598103 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f94bc021030 con 0x7f94c40fc870 2024-01-19T01:16:05.390 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:05.390 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:05.392 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f94ac0366d0 msgr2=0x7f94ac038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:05.392 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f94ac0366d0 0x7f94ac038b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009990 tx=0x7f94b8006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:05.392 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 msgr2=0x7f94c410cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:05.392 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c410cba0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f94bc000f80 tx=0x7f94bc004d10 comp rx=0 tx=0).stop 2024-01-19T01:16:05.392 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 shutdown_connections 2024-01-19T01:16:05.392 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f94ac0366d0 0x7f94ac038b80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:05.392 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 --2- 172.21.15.152:0/3769598103 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f94c40fc870 0x7f94c410cba0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:05.393 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 >> 172.21.15.152:0/3769598103 conn(0x7f94c40f7f50 msgr2=0x7f94c40fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:05.393 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 shutdown_connections 2024-01-19T01:16:05.393 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:05.391+0000 7f94c8cf6700 1 -- 172.21.15.152:0/3769598103 wait complete. 2024-01-19T01:16:05.393 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:05.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:05 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/3769598103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:06 smithi079 ceph-mon[28374]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:06.870 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:06.871 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:07.125 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:07.172 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:08.550 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.549+0000 7fa265f37700 1 -- 172.21.15.152:0/1213220641 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 msgr2=0x7fa2600fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:08.550 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.549+0000 7fa265f37700 1 --2- 172.21.15.152:0/1213220641 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa2600fbb80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa250009c60 tx=0x7fa250004d10 comp rx=0 tx=0).stop 2024-01-19T01:16:08.550 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.549+0000 7fa265f37700 1 -- 172.21.15.152:0/1213220641 shutdown_connections 2024-01-19T01:16:08.550 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.549+0000 7fa265f37700 1 --2- 172.21.15.152:0/1213220641 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa2600fbb80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:08.551 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.549+0000 7fa265f37700 1 -- 172.21.15.152:0/1213220641 >> 172.21.15.152:0/1213220641 conn(0x7fa2600f6e50 msgr2=0x7fa2600f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:08.551 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.549+0000 7fa265f37700 1 -- 172.21.15.152:0/1213220641 shutdown_connections 2024-01-19T01:16:08.551 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.549+0000 7fa265f37700 1 -- 172.21.15.152:0/1213220641 wait complete. 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.550+0000 7fa265f37700 1 Processor -- start 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.550+0000 7fa265f37700 1 -- start start 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.550+0000 7fa265f37700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa26010cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.550+0000 7fa265f37700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26010d0c0 con 0x7fa2600fb770 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.550+0000 7fa25f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa26010cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.550+0000 7fa25f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa26010cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:52308/0 (socket says 172.21.15.152:52308) 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.550+0000 7fa25f7fe700 1 -- 172.21.15.152:0/5132294 learned_addr learned my addr 172.21.15.152:0/5132294 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:08.552 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.551+0000 7fa25f7fe700 1 -- 172.21.15.152:0/5132294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa250009940 con 0x7fa2600fb770 2024-01-19T01:16:08.553 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.551+0000 7fa25f7fe700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa26010cb80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa250000f80 tx=0x7fa250004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:08.553 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.551+0000 7fa247fff700 1 -- 172.21.15.152:0/5132294 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa25000bec0 con 0x7fa2600fb770 2024-01-19T01:16:08.553 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.551+0000 7fa247fff700 1 -- 172.21.15.152:0/5132294 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa25001d8f0 con 0x7fa2600fb770 2024-01-19T01:16:08.553 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.551+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26010d2c0 con 0x7fa2600fb770 2024-01-19T01:16:08.553 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.551+0000 7fa247fff700 1 -- 172.21.15.152:0/5132294 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa250012890 con 0x7fa2600fb770 2024-01-19T01:16:08.553 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.551+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26010df60 con 0x7fa2600fb770 2024-01-19T01:16:08.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.552+0000 7fa247fff700 1 -- 172.21.15.152:0/5132294 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fa250019040 con 0x7fa2600fb770 2024-01-19T01:16:08.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.552+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26005f270 con 0x7fa2600fb770 2024-01-19T01:16:08.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.552+0000 7fa247fff700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fa2480366d0 0x7fa248038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:08.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.552+0000 7fa247fff700 1 -- 172.21.15.152:0/5132294 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa25004a200 con 0x7fa2600fb770 2024-01-19T01:16:08.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.552+0000 7fa25effd700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fa2480366d0 0x7fa248038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:08.554 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.553+0000 7fa25effd700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fa2480366d0 0x7fa248038b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa254009990 tx=0x7fa254006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:08.557 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.557+0000 7fa247fff700 1 -- 172.21.15.152:0/5132294 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa250026400 con 0x7fa2600fb770 2024-01-19T01:16:08.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:08 smithi079 ceph-mon[28374]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:08.819 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.818+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa26010ea50 con 0x7fa2600fb770 2024-01-19T01:16:08.820 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.819+0000 7fa247fff700 1 -- 172.21.15.152:0/5132294 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa250021030 con 0x7fa2600fb770 2024-01-19T01:16:08.820 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:08.820 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fa2480366d0 msgr2=0x7fa248038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fa2480366d0 0x7fa248038b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa254009990 tx=0x7fa254006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 msgr2=0x7fa26010cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa26010cb80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa250000f80 tx=0x7fa250004d10 comp rx=0 tx=0).stop 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 shutdown_connections 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7fa2480366d0 0x7fa248038b80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 --2- 172.21.15.152:0/5132294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa2600fb770 0x7fa26010cb80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 >> 172.21.15.152:0/5132294 conn(0x7fa2600f6e50 msgr2=0x7fa2600660c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 shutdown_connections 2024-01-19T01:16:08.822 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:08.821+0000 7fa265f37700 1 -- 172.21.15.152:0/5132294 wait complete. 2024-01-19T01:16:08.823 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:09.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:09 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/5132294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:10.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:10.398 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:10.647 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:10.658 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:10 smithi079 ceph-mon[28374]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:10.693 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:12.062 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 -- 172.21.15.152:0/3604665875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 msgr2=0x7f2d980fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 --2- 172.21.15.152:0/3604665875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d980fcc80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2d880048c0 tx=0x7f2d88009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 -- 172.21.15.152:0/3604665875 shutdown_connections 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 --2- 172.21.15.152:0/3604665875 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d980fcc80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 -- 172.21.15.152:0/3604665875 >> 172.21.15.152:0/3604665875 conn(0x7f2d980f7f50 msgr2=0x7f2d980fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 -- 172.21.15.152:0/3604665875 shutdown_connections 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 -- 172.21.15.152:0/3604665875 wait complete. 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.061+0000 7f2d9cc30700 1 Processor -- start 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.062+0000 7f2d9cc30700 1 -- start start 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.062+0000 7f2d9cc30700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d9810cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:12.063 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.062+0000 7f2d9cc30700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9810d020 con 0x7f2d980fc870 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.062+0000 7f2d9659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d9810cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.062+0000 7f2d9659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d9810cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:52330/0 (socket says 172.21.15.152:52330) 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.062+0000 7f2d9659c700 1 -- 172.21.15.152:0/1230847066 learned_addr learned my addr 172.21.15.152:0/1230847066 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.062+0000 7f2d9659c700 1 -- 172.21.15.152:0/1230847066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d88005040 con 0x7f2d980fc870 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.063+0000 7f2d9659c700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d9810cae0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009330 tx=0x7f2d88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.063+0000 7f2d86ffd700 1 -- 172.21.15.152:0/1230847066 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d880096b0 con 0x7f2d980fc870 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.063+0000 7f2d86ffd700 1 -- 172.21.15.152:0/1230847066 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d88009810 con 0x7f2d980fc870 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.063+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9810d220 con 0x7f2d980fc870 2024-01-19T01:16:12.065 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.063+0000 7f2d86ffd700 1 -- 172.21.15.152:0/1230847066 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8801c5b0 con 0x7f2d980fc870 2024-01-19T01:16:12.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.063+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9810dec0 con 0x7f2d980fc870 2024-01-19T01:16:12.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.064+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9805f270 con 0x7f2d980fc870 2024-01-19T01:16:12.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.064+0000 7f2d86ffd700 1 -- 172.21.15.152:0/1230847066 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f2d8801c870 con 0x7f2d980fc870 2024-01-19T01:16:12.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.064+0000 7f2d86ffd700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f2d800366d0 0x7f2d80038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:12.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.064+0000 7f2d86ffd700 1 -- 172.21.15.152:0/1230847066 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2d88049b60 con 0x7f2d980fc870 2024-01-19T01:16:12.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.064+0000 7f2d95d9b700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f2d800366d0 0x7f2d80038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:12.066 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.064+0000 7f2d95d9b700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f2d800366d0 0x7f2d80038b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009990 tx=0x7f2d8c006e40 comp rx=0 tx=0).ready entity=mgr.14158 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:12.069 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.069+0000 7f2d86ffd700 1 -- 172.21.15.152:0/1230847066 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2d88011730 con 0x7f2d980fc870 2024-01-19T01:16:12.327 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.325+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d9810ea10 con 0x7f2d980fc870 2024-01-19T01:16:12.327 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.326+0000 7f2d86ffd700 1 -- 172.21.15.152:0/1230847066 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2d88011370 con 0x7f2d980fc870 2024-01-19T01:16:12.327 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:12.327 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:12.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f2d800366d0 msgr2=0x7f2d80038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:12.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f2d800366d0 0x7f2d80038b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c009990 tx=0x7f2d8c006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:12.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 msgr2=0x7f2d9810cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:12.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d9810cae0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009330 tx=0x7f2d88009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:12.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 shutdown_connections 2024-01-19T01:16:12.329 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f2d800366d0 0x7f2d80038b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:12.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 --2- 172.21.15.152:0/1230847066 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d980fc870 0x7f2d9810cae0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:12.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 >> 172.21.15.152:0/1230847066 conn(0x7f2d980f7f50 msgr2=0x7f2d980f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:12.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 shutdown_connections 2024-01-19T01:16:12.330 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:12.328+0000 7f2d9cc30700 1 -- 172.21.15.152:0/1230847066 wait complete. 2024-01-19T01:16:12.330 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:12.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:12 smithi079 ceph-mon[28374]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:12.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:12 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:12.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:12 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:12.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:12 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:12.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:12 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-19T01:16:12.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:12 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/1230847066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:13.927 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:13.928 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:13.995 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:13 smithi079 ceph-mon[28374]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-19T01:16:13.995 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:13 smithi079 ceph-mon[28374]: mgrmap e14: smithi079.miqczk(active, since 63s) 2024-01-19T01:16:14.181 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:14.228 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:15.634 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 -- 172.21.15.152:0/536748919 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 msgr2=0x7f76540fbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:15.634 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 --2- 172.21.15.152:0/536748919 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540fbba0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f76500048c0 tx=0x7f7650009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:15.634 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 -- 172.21.15.152:0/536748919 shutdown_connections 2024-01-19T01:16:15.634 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 --2- 172.21.15.152:0/536748919 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540fbba0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:15.634 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 -- 172.21.15.152:0/536748919 >> 172.21.15.152:0/536748919 conn(0x7f76540f6e50 msgr2=0x7f76540f9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:15.635 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 -- 172.21.15.152:0/536748919 shutdown_connections 2024-01-19T01:16:15.635 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 -- 172.21.15.152:0/536748919 wait complete. 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 Processor -- start 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.633+0000 7f765b4f8700 1 -- start start 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f765b4f8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f765b4f8700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76540f7430 con 0x7f76540fb790 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7659294700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7659294700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:52350/0 (socket says 172.21.15.152:52350) 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7659294700 1 -- 172.21.15.152:0/2353130990 learned_addr learned my addr 172.21.15.152:0/2353130990 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7659294700 1 -- 172.21.15.152:0/2353130990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7650005040 con 0x7f76540fb790 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7659294700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540f8fb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7650009330 tx=0x7f7650009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:15.637 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7649ffb700 1 -- 172.21.15.152:0/2353130990 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76500096b0 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76540f7630 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7649ffb700 1 -- 172.21.15.152:0/2353130990 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7650009810 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.634+0000 7f7649ffb700 1 -- 172.21.15.152:0/2353130990 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765001c5b0 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.635+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76540f7b20 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.635+0000 7f7649ffb700 1 -- 172.21.15.152:0/2353130990 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f7650013420 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.635+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7654106430 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.636+0000 7f7649ffb700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 0x7f7640038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.636+0000 7f7649ffb700 1 -- 172.21.15.152:0/2353130990 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7650049b20 con 0x7f76540fb790 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.636+0000 7f7658a93700 1 -- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 msgr2=0x7f7640038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/4076006582 2024-01-19T01:16:15.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.636+0000 7f7658a93700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 0x7f7640038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T01:16:15.644 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.643+0000 7f7649ffb700 1 -- 172.21.15.152:0/2353130990 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f76500119c0 con 0x7f76540fb790 2024-01-19T01:16:15.838 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.837+0000 7f7658a93700 1 -- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 msgr2=0x7f7640038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/4076006582 2024-01-19T01:16:15.838 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.837+0000 7f7658a93700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 0x7f7640038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-19T01:16:15.899 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.898+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7654101410 con 0x7f76540fb790 2024-01-19T01:16:15.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.899+0000 7f7649ffb700 1 -- 172.21.15.152:0/2353130990 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7650011370 con 0x7f76540fb790 2024-01-19T01:16:15.900 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:15.900 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 msgr2=0x7f7640038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T01:16:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 0x7f7640038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 msgr2=0x7f76540f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540f8fb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7650009330 tx=0x7f7650009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 shutdown_connections 2024-01-19T01:16:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f7640036720 0x7f7640038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 --2- 172.21.15.152:0/2353130990 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f76540fb790 0x7f76540f8fb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 >> 172.21.15.152:0/2353130990 conn(0x7f76540f6e50 msgr2=0x7f7654066120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 shutdown_connections 2024-01-19T01:16:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:15.901+0000 7f765b4f8700 1 -- 172.21.15.152:0/2353130990 wait complete. 2024-01-19T01:16:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:16.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:15 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/2353130990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:17.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:17.416 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:17.664 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:17.712 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:19.096 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 -- 172.21.15.152:0/2640612380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 msgr2=0x7f75c40fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 --2- 172.21.15.152:0/2640612380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c40fcc80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f75b40048c0 tx=0x7f75b4009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 -- 172.21.15.152:0/2640612380 shutdown_connections 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 --2- 172.21.15.152:0/2640612380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c40fcc80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 -- 172.21.15.152:0/2640612380 >> 172.21.15.152:0/2640612380 conn(0x7f75c40f7f50 msgr2=0x7f75c40fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 -- 172.21.15.152:0/2640612380 shutdown_connections 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 -- 172.21.15.152:0/2640612380 wait complete. 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 Processor -- start 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 -- start start 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c410ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75cc3a0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c410cfd0 con 0x7f75c40fc870 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75ca13c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c410ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:19.097 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75ca13c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c410ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:58154/0 (socket says 172.21.15.152:58154) 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.094+0000 7f75ca13c700 1 -- 172.21.15.152:0/3968579163 learned_addr learned my addr 172.21.15.152:0/3968579163 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.095+0000 7f75ca13c700 1 -- 172.21.15.152:0/3968579163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b4005040 con 0x7f75c40fc870 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.095+0000 7f75ca13c700 1 --2- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c410ca90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009330 tx=0x7f75b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.095+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b40096b0 con 0x7f75c40fc870 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.095+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75b4009810 con 0x7f75c40fc870 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.095+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c410d1d0 con 0x7f75c40fc870 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.095+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75b401c5b0 con 0x7f75c40fc870 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.095+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c410de70 con 0x7f75c40fc870 2024-01-19T01:16:19.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.096+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f75b401c870 con 0x7f75c40fc870 2024-01-19T01:16:19.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.096+0000 7f75baffd700 1 --2- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f75b0036720 0x7f75b0038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:19.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.096+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f75b4049b40 con 0x7f75c40fc870 2024-01-19T01:16:19.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.096+0000 7f75c993b700 1 -- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f75b0036720 msgr2=0x7f75b0038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.79:6800/4076006582 2024-01-19T01:16:19.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.096+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c405f270 con 0x7f75c40fc870 2024-01-19T01:16:19.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.096+0000 7f75c993b700 1 --2- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f75b0036720 0x7f75b0038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T01:16:19.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.101+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f75b4011bd0 con 0x7f75c40fc870 2024-01-19T01:16:19.165 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.164+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mgrmap(e 15) v1 ==== 43075+0+0 (secure 0 0 0) 0x7f75b4049d50 con 0x7f75c40fc870 2024-01-19T01:16:19.165 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.164+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f75b0036720 msgr2=0x7f75b0038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T01:16:19.165 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.164+0000 7f75baffd700 1 --2- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f75b0036720 0x7f75b0038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:19.368 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.367+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75c410e920 con 0x7f75c40fc870 2024-01-19T01:16:19.368 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.367+0000 7f75baffd700 1 -- 172.21.15.152:0/3968579163 <== mon.0 v2:172.21.15.79:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f75b4021e60 con 0x7f75c40fc870 2024-01-19T01:16:19.368 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:19.369 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:19.370 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.369+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 msgr2=0x7f75c410ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.369+0000 7f75cc3a0700 1 --2- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c410ca90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009330 tx=0x7f75b4009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.369+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 shutdown_connections 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.369+0000 7f75cc3a0700 1 --2- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:6800/4076006582,v1:172.21.15.79:6801/4076006582] conn(0x7f75b0036720 0x7f75b0038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.369+0000 7f75cc3a0700 1 --2- 172.21.15.152:0/3968579163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75c40fc870 0x7f75c410ca90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.369+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 >> 172.21.15.152:0/3968579163 conn(0x7f75c40f7f50 msgr2=0x7f75c40f8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.370+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 shutdown_connections 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:19.370+0000 7f75cc3a0700 1 -- 172.21.15.152:0/3968579163 wait complete. 2024-01-19T01:16:19.371 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:19.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: Active manager daemon smithi079.miqczk restarted 2024-01-19T01:16:19.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: Activating manager daemon smithi079.miqczk 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: osdmap e5: 0 total, 0 up, 0 in 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: mgrmap e15: smithi079.miqczk(active, starting, since 0.00314919s) 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr metadata", "who": "smithi079.miqczk", "id": "smithi079.miqczk"}]: dispatch 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: Manager daemon smithi079.miqczk is now available 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:16:19.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:16:20.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:20 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:20.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:20 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:16:20.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:20 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/mirror_snapshot_schedule"}]: dispatch 2024-01-19T01:16:20.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:20 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/trash_purge_schedule"}]: dispatch 2024-01-19T01:16:20.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:20 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/3968579163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:20.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:20 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:20.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:20 smithi079 ceph-mon[28374]: mgrmap e16: smithi079.miqczk(active, since 1.00562s) 2024-01-19T01:16:20.895 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:20.895 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:21.154 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:21.202 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /etc/ceph/ceph.conf 2024-01-19T01:16:21.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:21 smithi079 ceph-mon[28374]: [19/Jan/2024:01:16:20] ENGINE Bus STARTING 2024-01-19T01:16:21.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:21 smithi079 ceph-mon[28374]: [19/Jan/2024:01:16:20] ENGINE Serving on https://172.21.15.79:7150 2024-01-19T01:16:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:21 smithi079 ceph-mon[28374]: [19/Jan/2024:01:16:20] ENGINE Bus STARTED 2024-01-19T01:16:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:21 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:21 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:21.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:21 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:23.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:22 smithi079 ceph-mon[28374]: mgrmap e17: smithi079.miqczk(active, since 2s) 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.583+0000 7f773d393700 1 -- 172.21.15.152:0/3364122413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 msgr2=0x7f77380fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.583+0000 7f773d393700 1 --2- 172.21.15.152:0/3364122413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f77380fbb90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f77200048c0 tx=0x7f7720009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.583+0000 7f773d393700 1 -- 172.21.15.152:0/3364122413 shutdown_connections 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.583+0000 7f773d393700 1 --2- 172.21.15.152:0/3364122413 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f77380fbb90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.583+0000 7f773d393700 1 -- 172.21.15.152:0/3364122413 >> 172.21.15.152:0/3364122413 conn(0x7f77380f6e40 msgr2=0x7f77380f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.583+0000 7f773d393700 1 -- 172.21.15.152:0/3364122413 shutdown_connections 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.583+0000 7f773d393700 1 -- 172.21.15.152:0/3364122413 wait complete. 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.584+0000 7f773d393700 1 Processor -- start 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.584+0000 7f773d393700 1 -- start start 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.584+0000 7f773d393700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f773810ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:23.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.584+0000 7f773d393700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773810cfc0 con 0x7f77380fb780 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.584+0000 7f7736ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f773810ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.584+0000 7f7736ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f773810ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:58176/0 (socket says 172.21.15.152:58176) 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.584+0000 7f7736ffd700 1 -- 172.21.15.152:0/3722781144 learned_addr learned my addr 172.21.15.152:0/3722781144 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.585+0000 7f7736ffd700 1 -- 172.21.15.152:0/3722781144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7720005040 con 0x7f77380fb780 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.585+0000 7f7736ffd700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f773810ca80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7720009330 tx=0x7f7720009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.585+0000 7f772f7fe700 1 -- 172.21.15.152:0/3722781144 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77200096b0 con 0x7f77380fb780 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.585+0000 7f772f7fe700 1 -- 172.21.15.152:0/3722781144 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7720009810 con 0x7f77380fb780 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.585+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773810d1c0 con 0x7f77380fb780 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.585+0000 7f772f7fe700 1 -- 172.21.15.152:0/3722781144 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772001c5b0 con 0x7f77380fb780 2024-01-19T01:16:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.585+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773810de60 con 0x7f77380fb780 2024-01-19T01:16:23.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.586+0000 7f772f7fe700 1 -- 172.21.15.152:0/3722781144 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f772001c870 con 0x7f77380fb780 2024-01-19T01:16:23.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.586+0000 7f772f7fe700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f77240367c0 0x7f7724038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:23.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.586+0000 7f772f7fe700 1 -- 172.21.15.152:0/3722781144 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7720049d40 con 0x7f77380fb780 2024-01-19T01:16:23.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.586+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773805f270 con 0x7f77380fb780 2024-01-19T01:16:23.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.586+0000 7f77367fc700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f77240367c0 0x7f7724038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:23.591 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.587+0000 7f77367fc700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f77240367c0 0x7f7724038c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7728009990 tx=0x7f7728006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:23.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.591+0000 7f772f7fe700 1 -- 172.21.15.152:0/3722781144 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7720023300 con 0x7f77380fb780 2024-01-19T01:16:23.859 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.858+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f773810e910 con 0x7f77380fb780 2024-01-19T01:16:23.859 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.858+0000 7f772f7fe700 1 -- 172.21.15.152:0/3722781144 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7720011b60 con 0x7f77380fb780 2024-01-19T01:16:23.859 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:23.859 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:23.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f77240367c0 msgr2=0x7f7724038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:23.861 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f77240367c0 0x7f7724038c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7728009990 tx=0x7f7728006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 msgr2=0x7f773810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f773810ca80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7720009330 tx=0x7f7720009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 shutdown_connections 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f77240367c0 0x7f7724038c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 --2- 172.21.15.152:0/3722781144 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f77380fb780 0x7f773810ca80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 >> 172.21.15.152:0/3722781144 conn(0x7f77380f6e40 msgr2=0x7f7738065610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 shutdown_connections 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:23.860+0000 7f773d393700 1 -- 172.21.15.152:0/3722781144 wait complete. 2024-01-19T01:16:23.862 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:24.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:24 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:24.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:24 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:24.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:24 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:24.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:24 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:24.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:24 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:16:24.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:24 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/3722781144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:25.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:25.653 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:25.905 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: Updating smithi079:/etc/ceph/ceph.conf 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: Updating smithi152:/etc/ceph/ceph.conf 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: Updating smithi079:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:26.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: Updating smithi152:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-19T01:16:26.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:27.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:27 smithi079 ceph-mon[28374]: Updating smithi079:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.client.admin.keyring 2024-01-19T01:16:27.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:27 smithi079 ceph-mon[28374]: Updating smithi152:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.client.admin.keyring 2024-01-19T01:16:27.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:27 smithi079 ceph-mon[28374]: Deploying daemon crash.smithi152 on smithi152 2024-01-19T01:16:28.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.724+0000 7f9d568a5700 1 -- 172.21.15.152:0/3428380533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 msgr2=0x7f9d50101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:28.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.724+0000 7f9d568a5700 1 --2- 172.21.15.152:0/3428380533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50101b80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9d440048c0 tx=0x7f9d44009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:28.728 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.724+0000 7f9d568a5700 1 -- 172.21.15.152:0/3428380533 shutdown_connections 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.724+0000 7f9d568a5700 1 --2- 172.21.15.152:0/3428380533 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50101b80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.724+0000 7f9d568a5700 1 -- 172.21.15.152:0/3428380533 >> 172.21.15.152:0/3428380533 conn(0x7f9d500f72f0 msgr2=0x7f9d500f9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.724+0000 7f9d568a5700 1 -- 172.21.15.152:0/3428380533 shutdown_connections 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.724+0000 7f9d568a5700 1 -- 172.21.15.152:0/3428380533 wait complete. 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.725+0000 7f9d568a5700 1 Processor -- start 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.725+0000 7f9d568a5700 1 -- start start 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.725+0000 7f9d568a5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50112b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.725+0000 7f9d568a5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50113080 con 0x7f9d501017b0 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50112b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50112b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:40238/0 (socket says 172.21.15.152:40238) 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4ffff700 1 -- 172.21.15.152:0/2099871314 learned_addr learned my addr 172.21.15.152:0/2099871314 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4ffff700 1 -- 172.21.15.152:0/2099871314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44005040 con 0x7f9d501017b0 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4ffff700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50112b40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009300 tx=0x7f9d44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:28.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4cff9700 1 -- 172.21.15.152:0/2099871314 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d44004e60 con 0x7f9d501017b0 2024-01-19T01:16:28.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d50113280 con 0x7f9d501017b0 2024-01-19T01:16:28.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5010ca00 con 0x7f9d501017b0 2024-01-19T01:16:28.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4cff9700 1 -- 172.21.15.152:0/2099871314 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d44013420 con 0x7f9d501017b0 2024-01-19T01:16:28.730 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.726+0000 7f9d4cff9700 1 -- 172.21.15.152:0/2099871314 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d4401c5b0 con 0x7f9d501017b0 2024-01-19T01:16:28.731 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.727+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c0052e0 con 0x7f9d501017b0 2024-01-19T01:16:28.731 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.727+0000 7f9d4cff9700 1 -- 172.21.15.152:0/2099871314 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f9d44009600 con 0x7f9d501017b0 2024-01-19T01:16:28.731 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.727+0000 7f9d4cff9700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9d38036810 0x7f9d38038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:28.731 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.727+0000 7f9d4cff9700 1 -- 172.21.15.152:0/2099871314 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9d44049800 con 0x7f9d501017b0 2024-01-19T01:16:28.731 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.728+0000 7f9d4f7fe700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9d38036810 0x7f9d38038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:28.731 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.728+0000 7f9d4f7fe700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9d38036810 0x7f9d38038cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009990 tx=0x7f9d40006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:28.734 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:28.732+0000 7f9d4cff9700 1 -- 172.21.15.152:0/2099871314 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9d44023300 con 0x7f9d501017b0 2024-01-19T01:16:29.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.015+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d3c005590 con 0x7f9d501017b0 2024-01-19T01:16:29.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.015+0000 7f9d4cff9700 1 -- 172.21.15.152:0/2099871314 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9d440116f0 con 0x7f9d501017b0 2024-01-19T01:16:29.017 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:29.018 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.017+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9d38036810 msgr2=0x7f9d38038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.017+0000 7f9d568a5700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9d38036810 0x7f9d38038cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009990 tx=0x7f9d40006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.017+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 msgr2=0x7f9d50112b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.017+0000 7f9d568a5700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50112b40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009300 tx=0x7f9d44009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.018+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 shutdown_connections 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.018+0000 7f9d568a5700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9d38036810 0x7f9d38038cc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.018+0000 7f9d568a5700 1 --2- 172.21.15.152:0/2099871314 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9d501017b0 0x7f9d50112b40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.018+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 >> 172.21.15.152:0/2099871314 conn(0x7f9d500f72f0 msgr2=0x7f9d500f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.018+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 shutdown_connections 2024-01-19T01:16:29.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:29.018+0000 7f9d568a5700 1 -- 172.21.15.152:0/2099871314 wait complete. 2024-01-19T01:16:29.021 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:29.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:29 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/2099871314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:30.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:30.653 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:30.928 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:31.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:31.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:31.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:31.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:31.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:31 smithi079 ceph-mon[28374]: Deploying daemon node-exporter.smithi152 on smithi152 2024-01-19T01:16:32.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.315+0000 7fce2e669700 1 -- 172.21.15.152:0/3812897294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 msgr2=0x7fce28101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:32.316 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.315+0000 7fce2e669700 1 --2- 172.21.15.152:0/3812897294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce28101bd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce18004940 tx=0x7fce18009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:32.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.315+0000 7fce2e669700 1 -- 172.21.15.152:0/3812897294 shutdown_connections 2024-01-19T01:16:32.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.315+0000 7fce2e669700 1 --2- 172.21.15.152:0/3812897294 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce28101bd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:32.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.315+0000 7fce2e669700 1 -- 172.21.15.152:0/3812897294 >> 172.21.15.152:0/3812897294 conn(0x7fce28071d60 msgr2=0x7fce28072160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:32.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.315+0000 7fce2e669700 1 -- 172.21.15.152:0/3812897294 shutdown_connections 2024-01-19T01:16:32.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.315+0000 7fce2e669700 1 -- 172.21.15.152:0/3812897294 wait complete. 2024-01-19T01:16:32.317 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.316+0000 7fce2e669700 1 Processor -- start 2024-01-19T01:16:32.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.316+0000 7fce2e669700 1 -- start start 2024-01-19T01:16:32.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.316+0000 7fce2e669700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce281108b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:32.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.316+0000 7fce2e669700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce28110df0 con 0x7fce28101800 2024-01-19T01:16:32.318 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.316+0000 7fce27fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce281108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.316+0000 7fce27fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce281108b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:40260/0 (socket says 172.21.15.152:40260) 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.316+0000 7fce27fff700 1 -- 172.21.15.152:0/86335455 learned_addr learned my addr 172.21.15.152:0/86335455 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.317+0000 7fce27fff700 1 -- 172.21.15.152:0/86335455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce18005040 con 0x7fce28101800 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.317+0000 7fce27fff700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce281108b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fce180099b0 tx=0x7fce18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.317+0000 7fce24ff9700 1 -- 172.21.15.152:0/86335455 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce18009230 con 0x7fce28101800 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.317+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce28110ff0 con 0x7fce28101800 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.317+0000 7fce24ff9700 1 -- 172.21.15.152:0/86335455 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce18009390 con 0x7fce28101800 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.317+0000 7fce24ff9700 1 -- 172.21.15.152:0/86335455 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce18024750 con 0x7fce28101800 2024-01-19T01:16:32.319 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.317+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2810a770 con 0x7fce28101800 2024-01-19T01:16:32.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.318+0000 7fce24ff9700 1 -- 172.21.15.152:0/86335455 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7fce18021070 con 0x7fce28101800 2024-01-19T01:16:32.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.318+0000 7fce24ff9700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fce100367c0 0x7fce10038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:32.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.318+0000 7fce24ff9700 1 -- 172.21.15.152:0/86335455 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fce1804d840 con 0x7fce28101800 2024-01-19T01:16:32.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.318+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2805f270 con 0x7fce28101800 2024-01-19T01:16:32.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.318+0000 7fce277fe700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fce100367c0 0x7fce10038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:32.320 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.319+0000 7fce277fe700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fce100367c0 0x7fce10038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009990 tx=0x7fce1c006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:32.324 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.323+0000 7fce24ff9700 1 -- 172.21.15.152:0/86335455 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fce1801b400 con 0x7fce28101800 2024-01-19T01:16:32.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.589+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce2804bc10 con 0x7fce28101800 2024-01-19T01:16:32.592 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.590+0000 7fce24ff9700 1 -- 172.21.15.152:0/86335455 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fce1802c390 con 0x7fce28101800 2024-01-19T01:16:32.592 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:32.592 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:32.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fce100367c0 msgr2=0x7fce10038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:32.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fce100367c0 0x7fce10038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fce1c009990 tx=0x7fce1c006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:32.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 msgr2=0x7fce281108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:32.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce281108b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fce180099b0 tx=0x7fce18009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:32.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 shutdown_connections 2024-01-19T01:16:32.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fce100367c0 0x7fce10038c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:32.593 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 --2- 172.21.15.152:0/86335455 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fce28101800 0x7fce281108b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:32.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 >> 172.21.15.152:0/86335455 conn(0x7fce28071d60 msgr2=0x7fce28102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:32.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 shutdown_connections 2024-01-19T01:16:32.594 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:32.592+0000 7fce2e669700 1 -- 172.21.15.152:0/86335455 wait complete. 2024-01-19T01:16:32.594 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:32.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:32 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/86335455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:34.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:34.389 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:34.855 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:35.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:35 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:35.500 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:35 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.689+0000 7f400ab41700 1 -- 172.21.15.152:0/2956458802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 msgr2=0x7f4004101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.689+0000 7f400ab41700 1 --2- 172.21.15.152:0/2956458802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004101bb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009c60 tx=0x7f3ffc004d10 comp rx=0 tx=0).stop 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.689+0000 7f400ab41700 1 -- 172.21.15.152:0/2956458802 shutdown_connections 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.689+0000 7f400ab41700 1 --2- 172.21.15.152:0/2956458802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004101bb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.689+0000 7f400ab41700 1 -- 172.21.15.152:0/2956458802 >> 172.21.15.152:0/2956458802 conn(0x7f40040715e0 msgr2=0x7f40040719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.689+0000 7f400ab41700 1 -- 172.21.15.152:0/2956458802 shutdown_connections 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.689+0000 7f400ab41700 1 -- 172.21.15.152:0/2956458802 wait complete. 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.690+0000 7f400ab41700 1 Processor -- start 2024-01-19T01:16:36.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.690+0000 7f400ab41700 1 -- start start 2024-01-19T01:16:36.692 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.690+0000 7f400ab41700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004112d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:36.692 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.690+0000 7f400ab41700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4004113290 con 0x7f40041017e0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.690+0000 7f40088dd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004112d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.690+0000 7f40088dd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004112d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:40268/0 (socket says 172.21.15.152:40268) 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.690+0000 7f40088dd700 1 -- 172.21.15.152:0/2282941412 learned_addr learned my addr 172.21.15.152:0/2282941412 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.691+0000 7f40088dd700 1 -- 172.21.15.152:0/2282941412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc009940 con 0x7f40041017e0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.691+0000 7f40088dd700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004112d50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009fd0 tx=0x7f3ffc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.691+0000 7f40017fa700 1 -- 172.21.15.152:0/2282941412 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc00be40 con 0x7f40041017e0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.691+0000 7f40017fa700 1 -- 172.21.15.152:0/2282941412 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ffc01d8f0 con 0x7f40041017e0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.691+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4004113490 con 0x7f40041017e0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.691+0000 7f40017fa700 1 -- 172.21.15.152:0/2282941412 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ffc012800 con 0x7f40041017e0 2024-01-19T01:16:36.693 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.691+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4004113900 con 0x7f40041017e0 2024-01-19T01:16:36.694 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.692+0000 7f40017fa700 1 -- 172.21.15.152:0/2282941412 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f3ffc019040 con 0x7f40041017e0 2024-01-19T01:16:36.694 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.692+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400405f270 con 0x7f40041017e0 2024-01-19T01:16:36.694 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.692+0000 7f40017fa700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3fec0367c0 0x7f3fec038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:36.694 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.692+0000 7f40017fa700 1 -- 172.21.15.152:0/2282941412 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3ffc04a1e0 con 0x7f40041017e0 2024-01-19T01:16:36.694 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.692+0000 7f4003fff700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3fec0367c0 0x7f3fec038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:36.695 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.693+0000 7f4003fff700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3fec0367c0 0x7f3fec038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009990 tx=0x7f3ff8006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:36.700 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.697+0000 7f40017fa700 1 -- 172.21.15.152:0/2282941412 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3ffc02eab0 con 0x7f40041017e0 2024-01-19T01:16:36.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:36.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:36 smithi079 ceph-mon[28374]: Deploying daemon mgr.smithi152.flyowb on smithi152 2024-01-19T01:16:36.966 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.965+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40040008d0 con 0x7f40041017e0 2024-01-19T01:16:36.966 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.965+0000 7f40017fa700 1 -- 172.21.15.152:0/2282941412 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3ffc021020 con 0x7f40041017e0 2024-01-19T01:16:36.966 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:36.967 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3fec0367c0 msgr2=0x7f3fec038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3fec0367c0 0x7f3fec038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009990 tx=0x7f3ff8006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 msgr2=0x7f4004112d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004112d50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009fd0 tx=0x7f3ffc004d10 comp rx=0 tx=0).stop 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 shutdown_connections 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3fec0367c0 0x7f3fec038c70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 --2- 172.21.15.152:0/2282941412 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40041017e0 0x7f4004112d50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 >> 172.21.15.152:0/2282941412 conn(0x7f40040715e0 msgr2=0x7f4004065610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 shutdown_connections 2024-01-19T01:16:36.969 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:36.968+0000 7f400ab41700 1 -- 172.21.15.152:0/2282941412 wait complete. 2024-01-19T01:16:36.970 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:37.691 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:37 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/2282941412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:39.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:39.581 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:40.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: Deploying daemon mon.smithi152 on smithi152 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:40.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:40.560 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:42.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:41 smithi152 systemd[1]: Starting Ceph mon.smithi152 for f3747e56-b667-11ee-95ae-87774f69a715... 2024-01-19T01:16:42.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:42 smithi079 ceph-mon[28374]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:42.950 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 -- 172.21.15.152:0/1967190566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 msgr2=0x7f23d4101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 --2- 172.21.15.152:0/1967190566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d4101ba0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f23d00048c0 tx=0x7f23d0009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 -- 172.21.15.152:0/1967190566 shutdown_connections 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 --2- 172.21.15.152:0/1967190566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d4101ba0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 -- 172.21.15.152:0/1967190566 >> 172.21.15.152:0/1967190566 conn(0x7f23d40f7310 msgr2=0x7f23d40f9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 -- 172.21.15.152:0/1967190566 shutdown_connections 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 -- 172.21.15.152:0/1967190566 wait complete. 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 Processor -- start 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.946+0000 7f23dbf62700 1 -- start start 2024-01-19T01:16:42.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.947+0000 7f23dbf62700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d410dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.947+0000 7f23dbf62700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d410e0f0 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.947+0000 7f23d9cfe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d410dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.947+0000 7f23d9cfe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d410dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.152:54312/0 (socket says 172.21.15.152:54312) 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.947+0000 7f23d9cfe700 1 -- 172.21.15.152:0/2974538579 learned_addr learned my addr 172.21.15.152:0/2974538579 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.947+0000 7f23d9cfe700 1 -- 172.21.15.152:0/2974538579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d0005040 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.947+0000 7f23d9cfe700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d410dbb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009300 tx=0x7f23d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.948+0000 7f23ca7fc700 1 -- 172.21.15.152:0/2974538579 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d0004e60 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.948+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d410e2f0 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.948+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d410e7e0 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.948+0000 7f23ca7fc700 1 -- 172.21.15.152:0/2974538579 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23d0013420 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.948+0000 7f23ca7fc700 1 -- 172.21.15.152:0/2974538579 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d00255c0 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.949+0000 7f23ca7fc700 1 -- 172.21.15.152:0/2974538579 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f23d0009600 con 0x7f23d41017d0 2024-01-19T01:16:42.996 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.949+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d4066680 con 0x7f23d41017d0 2024-01-19T01:16:42.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.949+0000 7f23ca7fc700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f23c0036770 0x7f23c0038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:42.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.949+0000 7f23ca7fc700 1 -- 172.21.15.152:0/2974538579 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f23d0049a20 con 0x7f23d41017d0 2024-01-19T01:16:42.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.949+0000 7f23d94fd700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f23c0036770 0x7f23c0038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:42.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.950+0000 7f23d94fd700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f23c0036770 0x7f23c0038c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009990 tx=0x7f23c4006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:42.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:42.954+0000 7f23ca7fc700 1 -- 172.21.15.152:0/2974538579 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f23d001c770 con 0x7f23d41017d0 2024-01-19T01:16:43.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 podman[35655]: 2024-01-19 01:16:43.009660198 +0000 UTC m=+0.017899452 image pull c3a232f785d83f930de30596257bc152e949b0a0d2045f8c153b4a5a153057bd quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d 2024-01-19T01:16:43.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 podman[35655]: 2024-01-19 01:16:43.109450778 +0000 UTC m=+0.117690023 container create 33474ce60d80c259c476442c8ed74926923c77cc2bddedb2e6a1755fa4982036 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi152, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-6441305, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240102, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD) 2024-01-19T01:16:43.240 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.238+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23d404bbb0 con 0x7f23d41017d0 2024-01-19T01:16:43.240 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.239+0000 7f23ca7fc700 1 -- 172.21.15.152:0/2974538579 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f23d0024360 con 0x7f23d41017d0 2024-01-19T01:16:43.241 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:43.241 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:14:04.956116Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-19T01:16:43.242 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f23c0036770 msgr2=0x7f23c0038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f23c0036770 0x7f23c0038c20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009990 tx=0x7f23c4006e40 comp rx=0 tx=0).stop 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 msgr2=0x7f23d410dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d410dbb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009300 tx=0x7f23d0009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 shutdown_connections 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f23c0036770 0x7f23c0038c20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 --2- 172.21.15.152:0/2974538579 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f23d41017d0 0x7f23d410dbb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 >> 172.21.15.152:0/2974538579 conn(0x7f23d40f7310 msgr2=0x7f23d40f92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:43.243 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 shutdown_connections 2024-01-19T01:16:43.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:43.241+0000 7f23dbf62700 1 -- 172.21.15.152:0/2974538579 wait complete. 2024-01-19T01:16:43.253 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-01-19T01:16:43.597 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 podman[35655]: 2024-01-19 01:16:43.356448994 +0000 UTC m=+0.364688239 container init 33474ce60d80c259c476442c8ed74926923c77cc2bddedb2e6a1755fa4982036 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi152, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240102, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-6441305, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-01-19T01:16:43.597 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 podman[35655]: 2024-01-19 01:16:43.360733076 +0000 UTC m=+0.368972325 container start 33474ce60d80c259c476442c8ed74926923c77cc2bddedb2e6a1755fa4982036 (image=quay.ceph.io/ceph-ci/ceph@sha256:1c6c19cb0c2f23b107e15db771fed3015fa219174b4acb3bb7355a8628f3e31d, name=ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi152, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, org.label-schema.build-date=20240102, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-6441305, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-01-19T01:16:43.597 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: set uid:gid to 167:167 (ceph:ceph) 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable), process ceph-mon, pid 2 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pidfile_write: ignore empty --pid-file 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: load: jerasure load: lrc 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: RocksDB version: 6.15.5 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Compile date Jan 9 2024 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: DB SUMMARY 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: DB Session ID: V1KAZU5PCSJU38J5KKKF 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: CURRENT file: CURRENT 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: IDENTITY file: IDENTITY 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi152/store.db dir, Total Num: 0, files: 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi152/store.db: 000004.log size: 511 ; 2024-01-19T01:16:43.598 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.error_if_exists: 0 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.create_if_missing: 0 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.paranoid_checks: 1 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.env: 0x557dae16aa80 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.fs: Posix File System 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.info_log: 0x557dafbe5000 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_file_opening_threads: 16 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.statistics: (nil) 2024-01-19T01:16:43.599 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.use_fsync: 0 2024-01-19T01:16:43.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_log_file_size: 0 2024-01-19T01:16:43.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-19T01:16:43.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-19T01:16:43.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.keep_log_file_num: 1000 2024-01-19T01:16:43.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.recycle_log_file_num: 0 2024-01-19T01:16:43.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.allow_fallocate: 1 2024-01-19T01:16:43.600 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.allow_mmap_reads: 0 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.allow_mmap_writes: 0 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.use_direct_reads: 0 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.create_missing_column_families: 0 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.db_log_dir: 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi152/store.db 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-19T01:16:43.601 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.advise_random_on_open: 1 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.db_write_buffer_size: 0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.write_buffer_manager: 0x557db095a9f0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.rate_limiter: (nil) 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.wal_recovery_mode: 2 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.enable_thread_tracking: 0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.enable_pipelined_write: 0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.unordered_write: 0 2024-01-19T01:16:43.602 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.row_cache: None 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.wal_filter: None 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.allow_ingest_behind: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.preserve_deletes: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.two_write_queues: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.manual_wal_flush: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.atomic_flush: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-19T01:16:43.603 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.log_readahead_size: 0 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.best_efforts_recovery: 0 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.allow_data_in_errors: 0 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.db_host_id: __hostname__ 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_background_jobs: 2 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_background_compactions: -1 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_subcompactions: 1 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-19T01:16:43.604 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_total_wal_size: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_open_files: -1 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bytes_per_sync: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_readahead_size: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_background_flushes: -1 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Compression algorithms supported: 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kZSTD supported: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kXpressCompression supported: 0 2024-01-19T01:16:43.605 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kLZ4HCCompression supported: 1 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kZlibCompression supported: 1 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kSnappyCompression supported: 1 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kLZ4Compression supported: 1 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: kBZip2Compression supported: 0 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi152/store.db/MANIFEST-000003 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.merge_operator: 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_filter: None 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_filter_factory: None 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.sst_partitioner_factory: None 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-19T01:16:43.606 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557dafc3af90) 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: cache_index_and_filter_blocks: 1 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: pin_top_level_index_and_filter: 1 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: index_type: 0 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: data_block_index_type: 0 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: index_shortening: 1 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: hash_index_allow_collision: 1 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: checksum: 1 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: no_block_cache: 0 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache: 0x557dafc6d090 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache_name: BinnedLRUCache 2024-01-19T01:16:43.607 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache_options: 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: capacity : 536870912 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: num_shard_bits : 4 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: strict_capacity_limit : 0 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: high_pri_pool_ratio: 0.000 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_cache_compressed: (nil) 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: persistent_cache: (nil) 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_size: 4096 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_size_deviation: 10 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_restart_interval: 16 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: index_block_restart_interval: 1 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: metadata_block_size: 4096 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: partition_filters: 0 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: use_delta_encoding: 1 2024-01-19T01:16:43.608 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: whole_key_filtering: 1 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: verify_compression: 0 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: read_amp_bytes_per_bit: 0 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: format_version: 4 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: enable_index_compression: 1 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: block_align: 0 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.write_buffer_size: 33554432 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_write_buffer_number: 2 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression: NoCompression 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression: Disabled 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.prefix_extractor: nullptr 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-19T01:16:43.609 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.num_levels: 7 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression_opts.level: 32767 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression_opts.strategy: 0 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-19T01:16:43.610 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compression_opts.enabled: false 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.target_file_size_base: 67108864 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-19T01:16:43.611 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-19T01:16:43.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-19T01:16:43.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-19T01:16:43.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-19T01:16:43.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-19T01:16:43.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.arena_block_size: 4194304 2024-01-19T01:16:43.612 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.disable_auto_compactions: 0 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-19T01:16:43.614 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.table_properties_collectors: 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.inplace_update_support: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.bloom_locality: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.max_successive_merges: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.paranoid_file_checks: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.force_consistency_checks: 1 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.report_bg_io_stats: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.ttl: 2592000 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-19T01:16:43.615 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.enable_blob_files: false 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.min_blob_size: 0 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.blob_file_size: 268435456 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi152/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705627003387019, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705627003387343, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705627003, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "0c14d1cc-b796-4305-a37b-4ba7708465d5", "db_session_id": "V1KAZU5PCSJU38J5KKKF"}} 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-01-19T01:16:43.616 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: EVENT_LOG_v1 {"time_micros": 1705627003387652, "job": 1, "event": "recovery_finished"} 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi152/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x557dafc01180 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: DB pointer 0x557dafcd6000 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** DB Stats ** 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** Compaction Stats [default] ** 2024-01-19T01:16:43.617 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** Compaction Stats [default] ** 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-19T01:16:43.618 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** Compaction Stats [default] ** 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-19T01:16:43.619 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** Compaction Stats [default] ** 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-01-19T01:16:43.620 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152 does not exist in monmap, will attempt to join an existing cluster 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: using public_addr v2:172.21.15.152:0/0 -> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: starting mon.smithi152 rank -1 at public addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] at bind addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi152 fsid f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(???) e0 preinit fsid f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).mds e1 new map 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).mds e1 print_map 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: e1 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: legacy client fscid: -1 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout: No filesystems configured 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-19T01:16:43.621 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Added host smithi152 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Deploying daemon alertmanager.smithi079 on smithi079 2024-01-19T01:16:43.622 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3808282809' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3808282809' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: osdmap e4: 0 total, 0 up, 0 in 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi079:172.21.15.79=smithi079;smithi152:172.21.15.152=smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Saving service mon spec with placement smithi079:172.21.15.79=smithi079;smithi152:172.21.15.152=smithi152;count:2 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.623 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Deploying daemon grafana.smithi079 on smithi079 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/1217706497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.624 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/728199680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/3087720042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/4233997397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Deploying daemon prometheus.smithi079 on smithi079 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/2547156959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.625 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/1660464073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/3769598103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/5132294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/1230847066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.626 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14158 172.21.15.79:0/2867849732' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mgrmap e14: smithi079.miqczk(active, since 63s) 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/2353130990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Active manager daemon smithi079.miqczk restarted 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Activating manager daemon smithi079.miqczk 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: osdmap e5: 0 total, 0 up, 0 in 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mgrmap e15: smithi079.miqczk(active, starting, since 0.00314919s) 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr metadata", "who": "smithi079.miqczk", "id": "smithi079.miqczk"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Manager daemon smithi079.miqczk is now available 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.627 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/mirror_snapshot_schedule"}]: dispatch 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi079.miqczk/trash_purge_schedule"}]: dispatch 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/3968579163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mgrmap e16: smithi079.miqczk(active, since 1.00562s) 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: [19/Jan/2024:01:16:20] ENGINE Bus STARTING 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: [19/Jan/2024:01:16:20] ENGINE Serving on https://172.21.15.79:7150 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: [19/Jan/2024:01:16:20] ENGINE Bus STARTED 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mgrmap e17: smithi079.miqczk(active, since 2s) 2024-01-19T01:16:43.628 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/3722781144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi079:/etc/ceph/ceph.conf 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi152:/etc/ceph/ceph.conf 2024-01-19T01:16:43.629 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi079:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi152:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi079:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.client.admin.keyring 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Updating smithi152:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.client.admin.keyring 2024-01-19T01:16:43.630 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Deploying daemon crash.smithi152 on smithi152 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/2099871314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Deploying daemon node-exporter.smithi152 on smithi152 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/86335455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.631 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Deploying daemon mgr.smithi152.flyowb on smithi152 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/2282941412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:43.632 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: Deploying daemon mon.smithi152 on smithi152 2024-01-19T01:16:43.633 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.633 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:43.633 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:43.633 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-01-19T01:16:43.633 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 ceph-mon[35694]: mon.smithi152@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-01-19T01:16:43.633 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 bash[35655]: 33474ce60d80c259c476442c8ed74926923c77cc2bddedb2e6a1755fa4982036 2024-01-19T01:16:43.633 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:43 smithi152 systemd[1]: Started Ceph mon.smithi152 for f3747e56-b667-11ee-95ae-87774f69a715. 2024-01-19T01:16:45.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-19T01:16:45.044 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-19T01:16:45.305 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: mon.smithi079 calling monitor election 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: mon.smithi152 calling monitor election 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.flyowb/crt"}]: dispatch 2024-01-19T01:16:48.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: mon.smithi079 is new leader, mons smithi079,smithi152 in quorum (ranks 0,1) 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: monmap e2: 2 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],smithi152=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: fsmap 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: osdmap e5: 0 total, 0 up, 0 in 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: mgrmap e17: smithi079.miqczk(active, since 29s) 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: Standby manager daemon smithi152.flyowb started 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: overall HEALTH_OK 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.flyowb/key"}]: dispatch 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-19T01:16:48.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:48.926 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:48.926 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:48.926 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:48 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: mon.smithi079 calling monitor election 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: mon.smithi152 calling monitor election 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.flyowb/crt"}]: dispatch 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:48.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: mon.smithi079 is new leader, mons smithi079,smithi152 in quorum (ranks 0,1) 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: monmap e2: 2 mons at {smithi079=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],smithi152=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: fsmap 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: osdmap e5: 0 total, 0 up, 0 in 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: mgrmap e17: smithi079.miqczk(active, since 29s) 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: Standby manager daemon smithi152.flyowb started 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: overall HEALTH_OK 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi152.flyowb/key"}]: dispatch 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.? 172.21.15.152:0/4157362028' entity='mgr.smithi152.flyowb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-19T01:16:49.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:49.001 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:49.001 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:49.001 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:48 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.597+0000 7fb384c4f700 1 -- 172.21.15.152:0/1078510395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 msgr2=0x7fb368005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.597+0000 7fb384c4f700 1 --2- 172.21.15.152:0/1078510395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 0x7fb368005a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb380065c50 tx=0x7fb37000b040 comp rx=0 tx=0).stop 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.597+0000 7fb384c4f700 1 -- 172.21.15.152:0/1078510395 shutdown_connections 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.597+0000 7fb384c4f700 1 --2- 172.21.15.152:0/1078510395 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 0x7fb368005a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.597+0000 7fb384c4f700 1 -- 172.21.15.152:0/1078510395 >> 172.21.15.152:0/1078510395 conn(0x7fb3800f73f0 msgr2=0x7fb3800f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.597+0000 7fb384c4f700 1 -- 172.21.15.152:0/1078510395 shutdown_connections 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.597+0000 7fb384c4f700 1 -- 172.21.15.152:0/1078510395 wait complete. 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.598+0000 7fb384c4f700 1 Processor -- start 2024-01-19T01:16:49.599 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.598+0000 7fb384c4f700 1 -- start start 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.598+0000 7fb384c4f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 0x7fb3800fb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.598+0000 7fb384c4f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 0x7fb3800fc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.598+0000 7fb384c4f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb380119060 con 0x7fb368003580 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.598+0000 7fb384c4f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3801191a0 con 0x7fb3800fbcf0 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37dd9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 0x7fb3800fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37dd9b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 0x7fb3800fc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:59824/0 (socket says 172.21.15.152:59824) 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37dd9b700 1 -- 172.21.15.152:0/1726615070 learned_addr learned my addr 172.21.15.152:0/1726615070 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37dd9b700 1 -- 172.21.15.152:0/1726615070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 msgr2=0x7fb3800fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37dd9b700 1 -- 172.21.15.152:0/1726615070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 msgr2=0x7fb3800fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37dd9b700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 0x7fb3800fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-19T01:16:49.601 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37e59c700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 0x7fb3800fb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37dd9b700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 0x7fb3800fc160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37e59c700 1 -- 172.21.15.152:0/1726615070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 msgr2=0x7fb3800fc160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37e59c700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 0x7fb3800fc160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37e59c700 1 -- 172.21.15.152:0/1726615070 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb370005040 con 0x7fb368003580 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb37e59c700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 0x7fb3800fb7b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb370004fd0 tx=0x7fb37000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.599+0000 7fb36effd700 1 -- 172.21.15.152:0/1726615070 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb370025030 con 0x7fb368003580 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.600+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3801193a0 con 0x7fb368003580 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.600+0000 7fb36effd700 1 -- 172.21.15.152:0/1726615070 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3700208f0 con 0x7fb368003580 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.600+0000 7fb36effd700 1 -- 172.21.15.152:0/1726615070 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb37001b030 con 0x7fb368003580 2024-01-19T01:16:49.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.600+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3801198b0 con 0x7fb368003580 2024-01-19T01:16:49.603 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.600+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38005f270 con 0x7fb368003580 2024-01-19T01:16:49.603 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.601+0000 7fb36effd700 1 -- 172.21.15.152:0/1726615070 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fb370028070 con 0x7fb368003580 2024-01-19T01:16:49.603 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.601+0000 7fb36effd700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb360068fe0 0x7fb36006b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:49.603 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.602+0000 7fb36effd700 1 -- 172.21.15.152:0/1726615070 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb370026070 con 0x7fb368003580 2024-01-19T01:16:49.603 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.602+0000 7fb37dd9b700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb360068fe0 0x7fb36006b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:49.603 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.602+0000 7fb37dd9b700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb360068fe0 0x7fb36006b490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb374006fd0 tx=0x7fb374008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:49.607 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.606+0000 7fb36effd700 1 -- 172.21.15.152:0/1726615070 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb3700549e0 con 0x7fb368003580 2024-01-19T01:16:49.900 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.899+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3801045b0 con 0x7fb368003580 2024-01-19T01:16:49.900 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:49 smithi152 ceph-mon[35694]: mgrmap e18: smithi079.miqczk(active, since 29s), standbys: smithi152.flyowb 2024-01-19T01:16:49.900 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr metadata", "who": "smithi152.flyowb", "id": "smithi152.flyowb"}]: dispatch 2024-01-19T01:16:49.901 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:16:49.901 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:16:49.901 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:49.901 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:49.901 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.899+0000 7fb36effd700 1 -- 172.21.15.152:0/1726615070 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7fb370058000 con 0x7fb368003580 2024-01-19T01:16:49.902 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-01-19T01:16:49.902 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":2,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","modified":"2024-01-19T01:16:43.413435Z","created":"2024-01-19T01:14:04.956116Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi079","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi152","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-19T01:16:49.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb360068fe0 msgr2=0x7fb36006b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:49.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb360068fe0 0x7fb36006b490 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb374006fd0 tx=0x7fb374008040 comp rx=0 tx=0).stop 2024-01-19T01:16:49.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 msgr2=0x7fb3800fb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:49.903 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 0x7fb3800fb7b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb370004fd0 tx=0x7fb37000b040 comp rx=0 tx=0).stop 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 shutdown_connections 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb360068fe0 0x7fb36006b490 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb368003580 0x7fb3800fb7b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 --2- 172.21.15.152:0/1726615070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3800fbcf0 0x7fb3800fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.901+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 >> 172.21.15.152:0/1726615070 conn(0x7fb3800f73f0 msgr2=0x7fb3800fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.902+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 shutdown_connections 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:16:49.902+0000 7fb384c4f700 1 -- 172.21.15.152:0/1726615070 wait complete. 2024-01-19T01:16:49.904 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 2 2024-01-19T01:16:49.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:49 smithi079 ceph-mon[28374]: mgrmap e18: smithi079.miqczk(active, since 29s), standbys: smithi152.flyowb 2024-01-19T01:16:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr metadata", "who": "smithi152.flyowb", "id": "smithi152.flyowb"}]: dispatch 2024-01-19T01:16:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:16:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:16:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:16:50.895 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:50 smithi152 ceph-mon[35694]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:50.895 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:50 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/1726615070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:50.917 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-19T01:16:50.917 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph config generate-minimal-conf 2024-01-19T01:16:50.948 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:50 smithi079 ceph-mon[28374]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:50.948 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:50 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/1726615070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-19T01:16:51.207 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: Updating smithi079:/etc/ceph/ceph.conf 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: Updating smithi152:/etc/ceph/ceph.conf 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: Updating smithi079:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: Updating smithi152:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: Reconfiguring mon.smithi079 (unknown last config time)... 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:52.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:51 smithi079 ceph-mon[28374]: Reconfiguring daemon mon.smithi079 on smithi079 2024-01-19T01:16:52.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: Updating smithi079:/etc/ceph/ceph.conf 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: Updating smithi152:/etc/ceph/ceph.conf 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: Updating smithi079:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: Updating smithi152:/var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/config/ceph.conf 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: Reconfiguring mon.smithi079 (unknown last config time)... 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:52.425 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:51 smithi152 ceph-mon[35694]: Reconfiguring daemon mon.smithi079 on smithi079 2024-01-19T01:16:53.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/904705569 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef00fbb30 msgr2=0x7f7ef00fbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:53.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/904705569 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef00fbb30 0x7f7ef00fbfa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0004940 tx=0x7f7ee0009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:53.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/904705569 shutdown_connections 2024-01-19T01:16:53.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/904705569 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef00fbb30 0x7f7ef00fbfa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:53.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/904705569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ef0101af0 0x7f7ef0101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:53.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/904705569 >> 172.21.15.79:0/904705569 conn(0x7f7ef00f7690 msgr2=0x7f7ef00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:53.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/904705569 shutdown_connections 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.927+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/904705569 wait complete. 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7ef4c8f700 1 Processor -- start 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7ef4c8f700 1 -- start start 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7ef4c8f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ef00fbb30 0x7f7ef010e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7ef4c8f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef0101af0 0x7f7ef010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7ef4c8f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef010f190 con 0x7f7ef0101af0 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7ef4c8f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef010f2d0 con 0x7f7ef00fbb30 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7eedd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef0101af0 0x7f7ef010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7eedd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef0101af0 0x7f7ef010ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:42018/0 (socket says 172.21.15.79:42018) 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7eedd9b700 1 -- 172.21.15.79:0/2577582840 learned_addr learned my addr 172.21.15.79:0/2577582840 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7eedd9b700 1 -- 172.21.15.79:0/2577582840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ef00fbb30 msgr2=0x7f7ef010e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:53.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7eedd9b700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ef00fbb30 0x7f7ef010e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:53.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7eedd9b700 1 -- 172.21.15.79:0/2577582840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee0005040 con 0x7f7ef0101af0 2024-01-19T01:16:53.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7eedd9b700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef0101af0 0x7f7ef010ec00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009fd0 tx=0x7f7ee0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:53.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.928+0000 7f7edeffd700 1 -- 172.21.15.79:0/2577582840 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ee000e040 con 0x7f7ef0101af0 2024-01-19T01:16:53.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.929+0000 7f7edeffd700 1 -- 172.21.15.79:0/2577582840 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ee000bc50 con 0x7f7ef0101af0 2024-01-19T01:16:53.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.929+0000 7f7edeffd700 1 -- 172.21.15.79:0/2577582840 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ee0013ea0 con 0x7f7ef0101af0 2024-01-19T01:16:53.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.929+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef01175d0 con 0x7f7ef0101af0 2024-01-19T01:16:53.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.929+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef0117ae0 con 0x7f7ef0101af0 2024-01-19T01:16:53.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.930+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef0103fd0 con 0x7f7ef0101af0 2024-01-19T01:16:53.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.930+0000 7f7edeffd700 1 -- 172.21.15.79:0/2577582840 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f7ee0013420 con 0x7f7ef0101af0 2024-01-19T01:16:53.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.930+0000 7f7edeffd700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7ed8068ed0 0x7f7ed806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:16:53.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.931+0000 7f7edeffd700 1 -- 172.21.15.79:0/2577582840 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7ee0087110 con 0x7f7ef0101af0 2024-01-19T01:16:53.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.931+0000 7f7eee59c700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7ed8068ed0 0x7f7ed806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:16:53.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.931+0000 7f7eee59c700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7ed8068ed0 0x7f7ed806b380 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4004d00 tx=0x7f7ee4009ce0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:16:53.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:53.935+0000 7f7edeffd700 1 -- 172.21.15.79:0/2577582840 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7ee00546c0 con 0x7f7ef0101af0 2024-01-19T01:16:54.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.125+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f7ef0064800 con 0x7f7ef0101af0 2024-01-19T01:16:54.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.125+0000 7f7edeffd700 1 -- 172.21.15.79:0/2577582840 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f7ee00868c0 con 0x7f7ef0101af0 2024-01-19T01:16:54.127 INFO:teuthology.orchestra.run.smithi079.stdout:# minimal ceph.conf for f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:16:54.127 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-01-19T01:16:54.127 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = f3747e56-b667-11ee-95ae-87774f69a715 2024-01-19T01:16:54.127 INFO:teuthology.orchestra.run.smithi079.stdout: mon_host = [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7ed8068ed0 msgr2=0x7f7ed806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7ed8068ed0 0x7f7ed806b380 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4004d00 tx=0x7f7ee4009ce0 comp rx=0 tx=0).stop 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef0101af0 msgr2=0x7f7ef010ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef0101af0 0x7f7ef010ec00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009fd0 tx=0x7f7ee0009e30 comp rx=0 tx=0).stop 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 shutdown_connections 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7ed8068ed0 0x7f7ed806b380 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ef00fbb30 0x7f7ef010e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 --2- 172.21.15.79:0/2577582840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ef0101af0 0x7f7ef010ec00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:16:54.129 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 >> 172.21.15.79:0/2577582840 conn(0x7f7ef00f7690 msgr2=0x7f7ef00f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:16:54.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 shutdown_connections 2024-01-19T01:16:54.130 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:16:54.127+0000 7f7ef4c8f700 1 -- 172.21.15.79:0/2577582840 wait complete. 2024-01-19T01:16:54.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: Reconfiguring mgr.smithi079.miqczk (unknown last config time)... 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi079.miqczk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: Reconfiguring daemon mgr.smithi079.miqczk on smithi079 2024-01-19T01:16:54.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:54 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2577582840' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:54.780 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-19T01:16:54.781 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:16:54.781 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-01-19T01:16:54.850 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:16:54.850 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-19T01:16:54.920 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:16:54.920 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-01-19T01:16:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: Reconfiguring mgr.smithi079.miqczk (unknown last config time)... 2024-01-19T01:16:54.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi079.miqczk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:16:54.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:16:54.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:54.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: Reconfiguring daemon mgr.smithi079.miqczk on smithi079 2024-01-19T01:16:54.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:54 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2577582840' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:54.955 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:16:54.955 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-19T01:16:55.026 INFO:tasks.cephadm:Deploying OSDs... 2024-01-19T01:16:55.026 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:16:55.026 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-01-19T01:16:55.042 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-19T01:16:55.042 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-01-19T01:16:55.096 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-19T01:16:55.096 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:55.097 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-01-19T01:16:55.097 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:55.097 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:55.097 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-01-19 01:16:10.987642039 +0000 2024-01-19T01:16:55.097 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-01-19 01:10:42.177537843 +0000 2024-01-19T01:16:55.097 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-01-19 01:10:42.177537843 +0000 2024-01-19T01:16:55.097 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-01-19 01:10:42.177537843 +0000 2024-01-19T01:16:55.097 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-19T01:16:55.166 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-01-19T01:16:55.166 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-01-19T01:16:55.166 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000201879 s, 2.5 MB/s 2024-01-19T01:16:55.168 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-19T01:16:55.224 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-01-19 01:16:10.987642039 +0000 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-01-19 01:10:42.507529933 +0000 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-01-19 01:10:42.507529933 +0000 2024-01-19T01:16:55.280 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-01-19 01:10:42.507529933 +0000 2024-01-19T01:16:55.281 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-19T01:16:55.345 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-01-19T01:16:55.345 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-01-19T01:16:55.345 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000189523 s, 2.7 MB/s 2024-01-19T01:16:55.346 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-19T01:16:55.402 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-01-19T01:16:55.457 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-19T01:16:55.458 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:55.458 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-19T01:16:55.458 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:55.458 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:55.458 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-01-19 01:16:10.988642015 +0000 2024-01-19T01:16:55.458 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-01-19 01:10:42.851521687 +0000 2024-01-19T01:16:55.458 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-01-19 01:10:42.851521687 +0000 2024-01-19T01:16:55.459 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-01-19 01:10:42.851521687 +0000 2024-01-19T01:16:55.459 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-19T01:16:55.524 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-01-19T01:16:55.524 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-01-19T01:16:55.524 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000196569 s, 2.6 MB/s 2024-01-19T01:16:55.526 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-19T01:16:55.581 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-01-19T01:16:55.638 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-19T01:16:55.638 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:55.639 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-01-19T01:16:55.639 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:55.639 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:55.639 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-01-19 01:16:10.988642015 +0000 2024-01-19T01:16:55.639 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-01-19 01:10:43.211513057 +0000 2024-01-19T01:16:55.639 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-01-19 01:10:43.211513057 +0000 2024-01-19T01:16:55.640 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-01-19 01:10:43.211513057 +0000 2024-01-19T01:16:55.640 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-19T01:16:55.705 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-01-19T01:16:55.705 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-01-19T01:16:55.705 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000211235 s, 2.4 MB/s 2024-01-19T01:16:55.706 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-19T01:16:55.762 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:16:55.763 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-01-19T01:16:55.782 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-19T01:16:55.783 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-01-19T01:16:55.840 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-19T01:16:55.840 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:55.840 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-01-19T01:16:55.840 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:55.840 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:55.841 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-01-19 01:16:41.867805051 +0000 2024-01-19T01:16:55.841 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-01-19 01:10:42.175799784 +0000 2024-01-19T01:16:55.841 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-01-19 01:10:42.175799784 +0000 2024-01-19T01:16:55.841 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-01-19 01:10:42.175799784 +0000 2024-01-19T01:16:55.841 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-19T01:16:55.909 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-01-19T01:16:55.909 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-01-19T01:16:55.909 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.00022848 s, 2.2 MB/s 2024-01-19T01:16:55.910 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-19T01:16:55.969 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-01-19 01:16:41.868805040 +0000 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-01-19 01:10:42.505796173 +0000 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-01-19 01:10:42.505796173 +0000 2024-01-19T01:16:56.026 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-01-19 01:10:42.505796173 +0000 2024-01-19T01:16:56.027 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-19T01:16:56.093 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-01-19T01:16:56.094 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-01-19T01:16:56.094 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000221031 s, 2.3 MB/s 2024-01-19T01:16:56.095 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-19T01:16:56.151 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-01-19T01:16:56.207 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-19T01:16:56.207 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:56.207 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-01-19T01:16:56.207 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:56.208 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:56.208 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-01-19 01:16:41.868805040 +0000 2024-01-19T01:16:56.208 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-01-19 01:10:42.829792627 +0000 2024-01-19T01:16:56.208 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-01-19 01:10:42.829792627 +0000 2024-01-19T01:16:56.208 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-01-19 01:10:42.829792627 +0000 2024-01-19T01:16:56.208 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-19T01:16:56.274 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-01-19T01:16:56.275 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-01-19T01:16:56.275 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000273296 s, 1.9 MB/s 2024-01-19T01:16:56.276 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-19T01:16:56.332 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-01-19 01:16:41.868805040 +0000 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-01-19 01:10:43.169788907 +0000 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-01-19 01:10:43.169788907 +0000 2024-01-19T01:16:56.388 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-01-19 01:10:43.169788907 +0000 2024-01-19T01:16:56.389 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-19T01:16:56.454 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-01-19T01:16:56.454 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-01-19T01:16:56.455 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000194964 s, 2.6 MB/s 2024-01-19T01:16:56.456 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-19T01:16:56.513 INFO:tasks.cephadm:Deploying osd.0 on smithi079 with /dev/vg_nvme/lv_4... 2024-01-19T01:16:56.513 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-19T01:16:56.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:56 smithi152 ceph-mon[35694]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:56.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:56.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:56.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:56 smithi152 ceph-mon[35694]: Reconfiguring crash.smithi079 (monmap changed)... 2024-01-19T01:16:56.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T01:16:56.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:56.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:56 smithi152 ceph-mon[35694]: Reconfiguring daemon crash.smithi079 on smithi079 2024-01-19T01:16:56.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:56 smithi079 ceph-mon[28374]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:56.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:56.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:56.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:56 smithi079 ceph-mon[28374]: Reconfiguring crash.smithi079 (monmap changed)... 2024-01-19T01:16:56.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi079", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T01:16:56.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:16:56.967 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:56 smithi079 ceph-mon[28374]: Reconfiguring daemon crash.smithi079 on smithi079 2024-01-19T01:16:57.102 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:16:58.315 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:58 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:58.316 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:58 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:58.316 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:58 smithi079 ceph-mon[28374]: Reconfiguring alertmanager.smithi079 (dependencies changed)... 2024-01-19T01:16:58.316 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:58 smithi079 ceph-mon[28374]: Reconfiguring daemon alertmanager.smithi079 on smithi079 2024-01-19T01:16:58.316 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:16:58 smithi079 ceph-mon[28374]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:58.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:58 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:58.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:58 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:16:58.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:58 smithi152 ceph-mon[35694]: Reconfiguring alertmanager.smithi079 (dependencies changed)... 2024-01-19T01:16:58.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:58 smithi152 ceph-mon[35694]: Reconfiguring daemon alertmanager.smithi079 on smithi079 2024-01-19T01:16:58.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:16:58 smithi152 ceph-mon[35694]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:16:59.631 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-19T01:16:59.632 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-19T01:16:59.632 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-01-19T01:16:59.632 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-01-19T01:16:59.632 INFO:teuthology.orchestra.run.smithi079.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113713 s, 922 MB/s 2024-01-19T01:16:59.632 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 2024-01-19T01:16:59.632 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-01-19T01:17:00.494 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2024-01-19T01:17:00.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:00 smithi152 ceph-mon[35694]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:00.747 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:00 smithi079 ceph-mon[28374]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:01.745 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:17:02.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:02 smithi079 ceph-mon[28374]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:02.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:02 smithi152 ceph-mon[35694]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:03.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.529+0000 7fe587673700 1 -- 172.21.15.79:0/3568864369 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fcd00 msgr2=0x7fe5800fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:03.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.529+0000 7fe587673700 1 --2- 172.21.15.79:0/3568864369 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fcd00 0x7fe5800fd150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe570004940 tx=0x7fe570009e30 comp rx=0 tx=0).stop 2024-01-19T01:17:03.533 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 -- 172.21.15.79:0/3568864369 shutdown_connections 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 --2- 172.21.15.79:0/3568864369 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fcd00 0x7fe5800fd150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 --2- 172.21.15.79:0/3568864369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5800fbaf0 0x7fe5800fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 -- 172.21.15.79:0/3568864369 >> 172.21.15.79:0/3568864369 conn(0x7fe5800f71d0 msgr2=0x7fe5800f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 -- 172.21.15.79:0/3568864369 shutdown_connections 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 -- 172.21.15.79:0/3568864369 wait complete. 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 Processor -- start 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 -- start start 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fbaf0 0x7fe5800ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5800fcd00 0x7fe5800fe040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800fe6d0 con 0x7fe5800fbaf0 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe587673700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800fe840 con 0x7fe5800fcd00 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe58540f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fbaf0 0x7fe5800ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe58540f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fbaf0 0x7fe5800ffbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49084/0 (socket says 172.21.15.79:49084) 2024-01-19T01:17:03.534 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.530+0000 7fe58540f700 1 -- 172.21.15.79:0/733054472 learned_addr learned my addr 172.21.15.79:0/733054472 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe584c0e700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5800fcd00 0x7fe5800fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe58540f700 1 -- 172.21.15.79:0/733054472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5800fcd00 msgr2=0x7fe5800fe040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe58540f700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5800fcd00 0x7fe5800fe040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe58540f700 1 -- 172.21.15.79:0/733054472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe570005040 con 0x7fe5800fbaf0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe584c0e700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5800fcd00 0x7fe5800fe040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe58540f700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fbaf0 0x7fe5800ffbc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00b790 tx=0x7fe57c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe575ffb700 1 -- 172.21.15.79:0/733054472 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe57c00f6f0 con 0x7fe5800fbaf0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe575ffb700 1 -- 172.21.15.79:0/733054472 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe57c00fd00 con 0x7fe5800fbaf0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe575ffb700 1 -- 172.21.15.79:0/733054472 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe57c0103a0 con 0x7fe5800fbaf0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5800feb20 con 0x7fe5800fbaf0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.531+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5800feea0 con 0x7fe5800fbaf0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.532+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58004bb80 con 0x7fe5800fbaf0 2024-01-19T01:17:03.535 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.532+0000 7fe575ffb700 1 -- 172.21.15.79:0/733054472 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fe57c00f850 con 0x7fe5800fbaf0 2024-01-19T01:17:03.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.533+0000 7fe575ffb700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe56c068ed0 0x7fe56c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:17:03.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.533+0000 7fe575ffb700 1 -- 172.21.15.79:0/733054472 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe57c085a10 con 0x7fe5800fbaf0 2024-01-19T01:17:03.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.533+0000 7fe584c0e700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe56c068ed0 0x7fe56c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:17:03.536 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.533+0000 7fe584c0e700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe56c068ed0 0x7fe56c06b380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe57000b3b0 tx=0x7fe570009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:17:03.538 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.537+0000 7fe575ffb700 1 -- 172.21.15.79:0/733054472 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe57c056a50 con 0x7fe5800fbaf0 2024-01-19T01:17:03.728 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:03.727+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe580066310 con 0x7fe56c068ed0 2024-01-19T01:17:04.532 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:04 smithi079 ceph-mon[28374]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:04.532 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:04 smithi079 ceph-mon[28374]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:17:04.533 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:17:04.533 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:17:04.533 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:04.533 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:04.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:04 smithi152 ceph-mon[35694]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:04 smithi152 ceph-mon[35694]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:17:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:17:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:17:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:06.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:06 smithi079 ceph-mon[28374]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:06.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:06 smithi152 ceph-mon[35694]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:06.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:07.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:07 smithi079 ceph-mon[28374]: Reconfiguring grafana.smithi079 (dependencies changed)... 2024-01-19T01:17:07.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:07 smithi079 ceph-mon[28374]: Reconfiguring daemon grafana.smithi079 on smithi079 2024-01-19T01:17:07.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:07 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/925114875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4500f84c-0d7c-472b-8d18-35de388ec5bb"}]: dispatch 2024-01-19T01:17:07.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:07 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/925114875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4500f84c-0d7c-472b-8d18-35de388ec5bb"}]': finished 2024-01-19T01:17:07.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:07 smithi079 ceph-mon[28374]: osdmap e6: 1 total, 0 up, 1 in 2024-01-19T01:17:07.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:07 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:07.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:07 smithi152 ceph-mon[35694]: Reconfiguring grafana.smithi079 (dependencies changed)... 2024-01-19T01:17:07.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:07 smithi152 ceph-mon[35694]: Reconfiguring daemon grafana.smithi079 on smithi079 2024-01-19T01:17:07.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:07 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/925114875' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4500f84c-0d7c-472b-8d18-35de388ec5bb"}]: dispatch 2024-01-19T01:17:07.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:07 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/925114875' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4500f84c-0d7c-472b-8d18-35de388ec5bb"}]': finished 2024-01-19T01:17:07.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:07 smithi152 ceph-mon[35694]: osdmap e6: 1 total, 0 up, 1 in 2024-01-19T01:17:07.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:07 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:08.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:08 smithi079 ceph-mon[28374]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:08.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:08 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2623473191' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:17:08.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:08 smithi152 ceph-mon[35694]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:08.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:08 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2623473191' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:17:10.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:10 smithi079 ceph-mon[28374]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:10.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:10 smithi152 ceph-mon[35694]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:12.543 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:12 smithi079 ceph-mon[28374]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:12.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:12 smithi152 ceph-mon[35694]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:14.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:14 smithi079 ceph-mon[28374]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:14 smithi152 ceph-mon[35694]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:16.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:16 smithi079 ceph-mon[28374]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:16.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:16.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:16 smithi152 ceph-mon[35694]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:17.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:17 smithi079 ceph-mon[28374]: Reconfiguring prometheus.smithi079 (dependencies changed)... 2024-01-19T01:17:17.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:17 smithi079 ceph-mon[28374]: Reconfiguring daemon prometheus.smithi079 on smithi079 2024-01-19T01:17:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:17 smithi152 ceph-mon[35694]: Reconfiguring prometheus.smithi079 (dependencies changed)... 2024-01-19T01:17:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:17 smithi152 ceph-mon[35694]: Reconfiguring daemon prometheus.smithi079 on smithi079 2024-01-19T01:17:18.678 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:18 smithi079 ceph-mon[28374]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:18.678 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:18 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-19T01:17:18.678 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:18 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:18.824 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:18 smithi152 ceph-mon[35694]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:18 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-19T01:17:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:18 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:19 smithi079 ceph-mon[28374]: Deploying daemon osd.0 on smithi079 2024-01-19T01:17:19.572 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:19.840 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:19 smithi152 ceph-mon[35694]: Deploying daemon osd.0 on smithi079 2024-01-19T01:17:19.840 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:20.672 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:20 smithi079 ceph-mon[28374]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:20.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:20 smithi152 ceph-mon[35694]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:22.477 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:22 smithi079 ceph-mon[28374]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:22.477 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:22.477 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:22.477 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T01:17:22.477 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:22.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:22 smithi152 ceph-mon[35694]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:22.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:22.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:22.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi152", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-19T01:17:22.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:23.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:23 smithi152 ceph-mon[35694]: Reconfiguring crash.smithi152 (monmap changed)... 2024-01-19T01:17:23.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:23 smithi152 ceph-mon[35694]: Reconfiguring daemon crash.smithi152 on smithi152 2024-01-19T01:17:23.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:23.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:23.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:17:23.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:17:23.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:23.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:23 smithi079 ceph-mon[28374]: Reconfiguring crash.smithi152 (monmap changed)... 2024-01-19T01:17:23.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:23 smithi079 ceph-mon[28374]: Reconfiguring daemon crash.smithi152 on smithi152 2024-01-19T01:17:23.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:23.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:23.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:17:23.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:17:23.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:24.673 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:24 smithi079 ceph-mon[28374]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:24.673 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:24 smithi079 ceph-mon[28374]: Reconfiguring mgr.smithi152.flyowb (monmap changed)... 2024-01-19T01:17:24.673 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:24 smithi079 ceph-mon[28374]: Reconfiguring daemon mgr.smithi152.flyowb on smithi152 2024-01-19T01:17:24.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:24 smithi152 ceph-mon[35694]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:24.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:24 smithi152 ceph-mon[35694]: Reconfiguring mgr.smithi152.flyowb (monmap changed)... 2024-01-19T01:17:24.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:24 smithi152 ceph-mon[35694]: Reconfiguring daemon mgr.smithi152.flyowb on smithi152 2024-01-19T01:17:25.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:25 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:25.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:25 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:25.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:25 smithi079 ceph-mon[28374]: Reconfiguring mon.smithi152 (monmap changed)... 2024-01-19T01:17:25.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:25 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T01:17:25.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:25 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T01:17:25.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:25 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:25.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:25 smithi079 ceph-mon[28374]: Reconfiguring daemon mon.smithi152 on smithi152 2024-01-19T01:17:25.940 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:25 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:25.940 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:25 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:25.940 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:25 smithi152 ceph-mon[35694]: Reconfiguring mon.smithi152 (monmap changed)... 2024-01-19T01:17:25.940 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:25 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-19T01:17:25.940 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:25 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-19T01:17:25.940 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:25 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:25.940 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:25 smithi152 ceph-mon[35694]: Reconfiguring daemon mon.smithi152 on smithi152 2024-01-19T01:17:26.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:26.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T01:17:26.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T01:17:26.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T01:17:26.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T01:17:26.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T01:17:26.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T01:17:26.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.826 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:17:26.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:27.941 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:27 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:28.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-19T01:17:28.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-01-19T01:17:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-19T01:17:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi079.front.sepia.ceph.com:3000"}]: dispatch 2024-01-19T01:17:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-01-19T01:17:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi079.front.sepia.ceph.com:9095"}]: dispatch 2024-01-19T01:17:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:27 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:28.936 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:28 smithi079 ceph-mon[28374]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:28.936 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:28 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:28.936 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:28 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:28 smithi152 ceph-mon[35694]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:28 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:28 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:30.156 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.154+0000 7fe575ffb700 1 -- 172.21.15.79:0/733054472 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe580066310 con 0x7fe56c068ed0 2024-01-19T01:17:30.156 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 0 on host 'smithi079' 2024-01-19T01:17:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe56c068ed0 msgr2=0x7fe56c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:30.157 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe56c068ed0 0x7fe56c06b380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe57000b3b0 tx=0x7fe570009e30 comp rx=0 tx=0).stop 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fbaf0 msgr2=0x7fe5800ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fbaf0 0x7fe5800ffbc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00b790 tx=0x7fe57c009220 comp rx=0 tx=0).stop 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 shutdown_connections 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe56c068ed0 0x7fe56c06b380 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5800fbaf0 0x7fe5800ffbc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.156+0000 7fe587673700 1 --2- 172.21.15.79:0/733054472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5800fcd00 0x7fe5800fe040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.157+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 >> 172.21.15.79:0/733054472 conn(0x7fe5800f71d0 msgr2=0x7fe580065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.157+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 shutdown_connections 2024-01-19T01:17:30.158 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:30.157+0000 7fe587673700 1 -- 172.21.15.79:0/733054472 wait complete. 2024-01-19T01:17:30.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:30 smithi152 ceph-mon[35694]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:30.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:30.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:30.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:30 smithi079 ceph-mon[28374]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:30.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:30.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:30.971 DEBUG:teuthology.orchestra.run.smithi079:osd.0> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@osd.0.service 2024-01-19T01:17:30.974 INFO:tasks.cephadm:Deploying osd.1 on smithi079 with /dev/vg_nvme/lv_3... 2024-01-19T01:17:30.974 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-19T01:17:31.774 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:17:31.979 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:31 smithi079 ceph-mon[28374]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-19T01:17:32.176 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:31 smithi152 ceph-mon[35694]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-19T01:17:32.859 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:32 smithi079 ceph-mon[28374]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:32.859 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:32 smithi079 ceph-mon[28374]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-19T01:17:32.859 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:32 smithi079 ceph-mon[28374]: osdmap e7: 1 total, 0 up, 1 in 2024-01-19T01:17:32.859 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:32.859 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:32 smithi079 ceph-mon[28374]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-01-19T01:17:32.859 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:17:32 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-0[46545]: 2024-01-19T01:17:32.717+0000 7fd52adc8700 -1 osd.0 0 waiting for initial osdmap 2024-01-19T01:17:32.859 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:17:32 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-0[46545]: 2024-01-19T01:17:32.721+0000 7fd5243c0700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T01:17:33.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:32 smithi152 ceph-mon[35694]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:33.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:32 smithi152 ceph-mon[35694]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-19T01:17:33.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:32 smithi152 ceph-mon[35694]: osdmap e7: 1 total, 0 up, 1 in 2024-01-19T01:17:33.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:33.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:32 smithi152 ceph-mon[35694]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-01-19T01:17:33.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:33 smithi079 ceph-mon[28374]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-01-19T01:17:33.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:33 smithi079 ceph-mon[28374]: osdmap e8: 1 total, 0 up, 1 in 2024-01-19T01:17:33.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:33 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:33.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:33 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:34.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:33 smithi152 ceph-mon[35694]: from='osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-01-19T01:17:34.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:33 smithi152 ceph-mon[35694]: osdmap e8: 1 total, 0 up, 1 in 2024-01-19T01:17:34.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:33 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:34.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:33 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:34.993 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: purged_snaps scrub starts 2024-01-19T01:17:34.993 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: purged_snaps scrub ok 2024-01-19T01:17:34.994 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:34.994 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] boot 2024-01-19T01:17:34.994 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: osdmap e9: 1 total, 1 up, 1 in 2024-01-19T01:17:34.994 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:34.994 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:34.994 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:34.995 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:34.995 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-19T01:17:34.995 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-19T01:17:34.995 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-01-19T01:17:34.995 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-01-19T01:17:34.996 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139755 s, 750 MB/s 2024-01-19T01:17:34.996 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-01-19T01:17:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: purged_snaps scrub starts 2024-01-19T01:17:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: purged_snaps scrub ok 2024-01-19T01:17:35.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-19T01:17:35.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: osd.0 [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] boot 2024-01-19T01:17:35.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: osdmap e9: 1 total, 1 up, 1 in 2024-01-19T01:17:35.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-19T01:17:35.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:35.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:35.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:35.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:35 smithi152 ceph-mon[35694]: osdmap e10: 1 total, 1 up, 1 in 2024-01-19T01:17:35.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:35 smithi079 ceph-mon[28374]: osdmap e10: 1 total, 1 up, 1 in 2024-01-19T01:17:36.221 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2024-01-19T01:17:36.914 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:36 smithi079 ceph-mon[28374]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:37.147 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:17:37.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:36 smithi152 ceph-mon[35694]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:38.909 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:38.909 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:38.909 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:38.909 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:17:38.910 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:38.910 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:38.910 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:17:38.910 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:38.910 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:17:39.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:39.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:39.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:39.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:17:39.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:39.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:39.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:17:39.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:39.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:17:39.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 -- 172.21.15.79:0/2140390104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 msgr2=0x7f5c5c0fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:39.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 --2- 172.21.15.79:0/2140390104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c0fd150 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c004940 tx=0x7f5c4c009e30 comp rx=0 tx=0).stop 2024-01-19T01:17:39.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 -- 172.21.15.79:0/2140390104 shutdown_connections 2024-01-19T01:17:39.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 --2- 172.21.15.79:0/2140390104 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c0fd150 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:39.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 --2- 172.21.15.79:0/2140390104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c5c0fbaf0 0x7f5c5c0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 -- 172.21.15.79:0/2140390104 >> 172.21.15.79:0/2140390104 conn(0x7f5c5c0f71b0 msgr2=0x7f5c5c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 -- 172.21.15.79:0/2140390104 shutdown_connections 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.405+0000 7f5c632b5700 1 -- 172.21.15.79:0/2140390104 wait complete. 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c632b5700 1 Processor -- start 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c632b5700 1 -- start start 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c632b5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c5c0fbaf0 0x7f5c5c10c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c632b5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c10c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c632b5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c10cec0 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c632b5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c10d760 con 0x7f5c5c0fbaf0 2024-01-19T01:17:39.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c60850700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c10c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c61051700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c5c0fbaf0 0x7f5c5c10c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c60850700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c10c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:35302/0 (socket says 172.21.15.79:35302) 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c60850700 1 -- 172.21.15.79:0/1861317340 learned_addr learned my addr 172.21.15.79:0/1861317340 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c60850700 1 -- 172.21.15.79:0/1861317340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c5c0fbaf0 msgr2=0x7f5c5c10c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c60850700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c5c0fbaf0 0x7f5c5c10c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c60850700 1 -- 172.21.15.79:0/1861317340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c005040 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.406+0000 7f5c61051700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c5c0fbaf0 0x7f5c5c10c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.407+0000 7f5c60850700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c10c8c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c000c00 tx=0x7f5c4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.407+0000 7f5c51ffb700 1 -- 172.21.15.79:0/1861317340 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c4c021070 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.407+0000 7f5c51ffb700 1 -- 172.21.15.79:0/1861317340 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c4c01c400 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.407+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c10d9e0 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.407+0000 7f5c51ffb700 1 -- 172.21.15.79:0/1861317340 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c4c00e070 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.407+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c10dec0 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.408+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c04bb80 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.409+0000 7f5c51ffb700 1 -- 172.21.15.79:0/1861317340 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f5c4c01c9c0 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.409+0000 7f5c51ffb700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5c48068ed0 0x7f5c4806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:17:39.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.409+0000 7f5c51ffb700 1 -- 172.21.15.79:0/1861317340 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f5c4c0871e0 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.409+0000 7f5c61051700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5c48068ed0 0x7f5c4806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:17:39.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.410+0000 7f5c61051700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5c48068ed0 0x7f5c4806b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009b80 tx=0x7f5c5800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:17:39.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.414+0000 7f5c51ffb700 1 -- 172.21.15.79:0/1861317340 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5c4c057f20 con 0x7f5c5c0fcd00 2024-01-19T01:17:39.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:39.610+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c5c10e390 con 0x7f5c48068ed0 2024-01-19T01:17:39.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:39 smithi079 ceph-mon[28374]: Detected new or changed devices on smithi079 2024-01-19T01:17:39.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:39 smithi079 ceph-mon[28374]: Adjusting osd_memory_target on smithi079 to 12864M 2024-01-19T01:17:39.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:39 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:17:39.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:39 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:17:39.865 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:39 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:40.093 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:39 smithi152 ceph-mon[35694]: Detected new or changed devices on smithi079 2024-01-19T01:17:40.093 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:39 smithi152 ceph-mon[35694]: Adjusting osd_memory_target on smithi079 to 12864M 2024-01-19T01:17:40.094 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:39 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:17:40.094 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:39 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:17:40.094 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:39 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:40.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:40 smithi079 ceph-mon[28374]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:40.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:40 smithi079 ceph-mon[28374]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:17:40.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:40.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:41.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:40 smithi152 ceph-mon[35694]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:41.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:40 smithi152 ceph-mon[35694]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:17:41.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:40 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:41.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:40 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:41.858 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:41 smithi079 ceph-mon[28374]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:42.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:41 smithi152 ceph-mon[35694]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:43.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:43 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1599789134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0286a58d-91d9-48f3-9a73-cad13c064623"}]: dispatch 2024-01-19T01:17:43.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:43 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1599789134' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0286a58d-91d9-48f3-9a73-cad13c064623"}]': finished 2024-01-19T01:17:43.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:43 smithi079 ceph-mon[28374]: osdmap e11: 2 total, 1 up, 2 in 2024-01-19T01:17:43.250 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:43 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:17:43.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1599789134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0286a58d-91d9-48f3-9a73-cad13c064623"}]: dispatch 2024-01-19T01:17:43.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1599789134' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0286a58d-91d9-48f3-9a73-cad13c064623"}]': finished 2024-01-19T01:17:43.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:43 smithi152 ceph-mon[35694]: osdmap e11: 2 total, 1 up, 2 in 2024-01-19T01:17:43.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:17:44.321 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:44 smithi079 ceph-mon[28374]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:44.321 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:44 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/195199354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:17:44.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:44 smithi152 ceph-mon[35694]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:44.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:44 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/195199354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:17:46.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:46.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:46.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:46.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:17:46.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:46.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:45 smithi152 ceph-mon[35694]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:46.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:46.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:46.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:46.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:17:46.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:46.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:45 smithi079 ceph-mon[28374]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:48.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:48 smithi079 ceph-mon[28374]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:48.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:48 smithi152 ceph-mon[35694]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:49.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:49.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:17:50.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:50 smithi079 ceph-mon[28374]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:50.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:50 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-19T01:17:50.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:50 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:50.846 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:50 smithi152 ceph-mon[35694]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:50.846 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:50 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-19T01:17:50.846 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:50 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:17:51.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:51 smithi079 ceph-mon[28374]: Deploying daemon osd.1 on smithi079 2024-01-19T01:17:51.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:51 smithi152 ceph-mon[35694]: Deploying daemon osd.1 on smithi079 2024-01-19T01:17:52.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:52 smithi079 ceph-mon[28374]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:52.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:52 smithi152 ceph-mon[35694]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:54.577 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:54 smithi079 ceph-mon[28374]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:54 smithi152 ceph-mon[35694]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:56.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:56 smithi079 ceph-mon[28374]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:56.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:56.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:17:56.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:56.789 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:56 smithi152 ceph-mon[35694]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:56.789 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:56.789 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:17:56.789 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:58.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:58 smithi079 ceph-mon[28374]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:58.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:58 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:58.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:17:58 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:58 smithi152 ceph-mon[35694]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:17:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:58 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:58.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:17:58 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:17:59.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.030+0000 7f5c51ffb700 1 -- 172.21.15.79:0/1861317340 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5c5c10e390 con 0x7f5c48068ed0 2024-01-19T01:17:59.031 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 1 on host 'smithi079' 2024-01-19T01:17:59.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5c48068ed0 msgr2=0x7f5c4806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:59.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5c48068ed0 0x7f5c4806b380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009b80 tx=0x7f5c5800b040 comp rx=0 tx=0).stop 2024-01-19T01:17:59.033 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 msgr2=0x7f5c5c10c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c10c8c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c000c00 tx=0x7f5c4c009e30 comp rx=0 tx=0).stop 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 shutdown_connections 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5c48068ed0 0x7f5c4806b380 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c5c0fbaf0 0x7f5c5c10c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 --2- 172.21.15.79:0/1861317340 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5c5c0fcd00 0x7f5c5c10c8c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 >> 172.21.15.79:0/1861317340 conn(0x7f5c5c0f71b0 msgr2=0x7f5c5c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 shutdown_connections 2024-01-19T01:17:59.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:17:59.032+0000 7f5c632b5700 1 -- 172.21.15.79:0/1861317340 wait complete. 2024-01-19T01:17:59.964 DEBUG:teuthology.orchestra.run.smithi079:osd.1> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@osd.1.service 2024-01-19T01:17:59.966 INFO:tasks.cephadm:Deploying osd.2 on smithi079 with /dev/vg_nvme/lv_2... 2024-01-19T01:17:59.966 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-19T01:18:00.242 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:00 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:00.243 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:00 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:00.243 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:00 smithi079 ceph-mon[28374]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:18:00.243 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:00 smithi079 ceph-mon[28374]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-19T01:18:00.356 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:00 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:00.356 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:00 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:00.356 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:00 smithi152 ceph-mon[35694]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:18:00.356 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:00 smithi152 ceph-mon[35694]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-19T01:18:00.774 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:18:01.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:01 smithi152 ceph-mon[35694]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-19T01:18:01.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:01 smithi152 ceph-mon[35694]: osdmap e12: 2 total, 1 up, 2 in 2024-01-19T01:18:01.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:01.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:01 smithi152 ceph-mon[35694]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-01-19T01:18:01.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:01 smithi079 ceph-mon[28374]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-19T01:18:01.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:01 smithi079 ceph-mon[28374]: osdmap e12: 2 total, 1 up, 2 in 2024-01-19T01:18:01.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:01.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:01 smithi079 ceph-mon[28374]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-01-19T01:18:01.249 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:01 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-1[53907]: 2024-01-19T01:18:01.013+0000 7fa1fcb4f700 -1 osd.1 0 waiting for initial osdmap 2024-01-19T01:18:01.249 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:01 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-1[53907]: 2024-01-19T01:18:01.016+0000 7fa1f6948700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T01:18:02.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:02 smithi079 ceph-mon[28374]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-01-19T01:18:02.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:02 smithi079 ceph-mon[28374]: osdmap e13: 2 total, 1 up, 2 in 2024-01-19T01:18:02.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:02.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:02.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:02 smithi079 ceph-mon[28374]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:18:02.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:02 smithi152 ceph-mon[35694]: from='osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-01-19T01:18:02.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:02 smithi152 ceph-mon[35694]: osdmap e13: 2 total, 1 up, 2 in 2024-01-19T01:18:02.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:02.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:02.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:02 smithi152 ceph-mon[35694]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-01-19T01:18:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-19T01:18:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-19T01:18:03.145 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10+0 records in 2024-01-19T01:18:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:10+0 records out 2024-01-19T01:18:03.145 INFO:teuthology.orchestra.run.smithi079.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123599 s, 848 MB/s 2024-01-19T01:18:03.145 INFO:teuthology.orchestra.run.smithi079.stderr:--> Zapping successful for: 2024-01-19T01:18:03.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:03 smithi152 ceph-mon[35694]: purged_snaps scrub starts 2024-01-19T01:18:03.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:03 smithi152 ceph-mon[35694]: purged_snaps scrub ok 2024-01-19T01:18:03.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:03 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:03.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:03 smithi152 ceph-mon[35694]: osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] boot 2024-01-19T01:18:03.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:03 smithi152 ceph-mon[35694]: osdmap e14: 2 total, 2 up, 2 in 2024-01-19T01:18:03.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:03 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:03.489 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:03 smithi079 ceph-mon[28374]: purged_snaps scrub starts 2024-01-19T01:18:03.489 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:03 smithi079 ceph-mon[28374]: purged_snaps scrub ok 2024-01-19T01:18:03.489 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:03 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:03.489 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:03 smithi079 ceph-mon[28374]: osd.1 [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] boot 2024-01-19T01:18:03.489 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:03 smithi079 ceph-mon[28374]: osdmap e14: 2 total, 2 up, 2 in 2024-01-19T01:18:03.489 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:03 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-19T01:18:03.833 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2024-01-19T01:18:04.228 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:18:04.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:04 smithi079 ceph-mon[28374]: osdmap e15: 2 total, 2 up, 2 in 2024-01-19T01:18:04.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:04 smithi079 ceph-mon[28374]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:04.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:04 smithi152 ceph-mon[35694]: osdmap e15: 2 total, 2 up, 2 in 2024-01-19T01:18:04.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:04 smithi152 ceph-mon[35694]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:05.307 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:05 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:05.307 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:05 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:05.308 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:05 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:05.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:05 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:05.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:05 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:05.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:05 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:06.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:06 smithi079 ceph-mon[28374]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:06.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:06 smithi152 ceph-mon[35694]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 -- 172.21.15.79:0/3237451882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 msgr2=0x7f5ab80fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/3237451882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab80fbf00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8004940 tx=0x7f5aa8009e30 comp rx=0 tx=0).stop 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 -- 172.21.15.79:0/3237451882 shutdown_connections 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/3237451882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ab80fcd00 0x7f5ab80fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/3237451882 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab80fbf00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 -- 172.21.15.79:0/3237451882 >> 172.21.15.79:0/3237451882 conn(0x7f5ab80f7190 msgr2=0x7f5ab80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 -- 172.21.15.79:0/3237451882 shutdown_connections 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.824+0000 7f5abcd7e700 1 -- 172.21.15.79:0/3237451882 wait complete. 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5abcd7e700 1 Processor -- start 2024-01-19T01:18:06.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5abcd7e700 1 -- start start 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5abcd7e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab810c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5abcd7e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ab80fcd00 0x7f5ab810ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5abcd7e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab810d0b0 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5abcd7e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab810d8f0 con 0x7f5ab80fcd00 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5ab659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab810c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5ab659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab810c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56436/0 (socket says 172.21.15.79:56436) 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5ab659c700 1 -- 172.21.15.79:0/692054155 learned_addr learned my addr 172.21.15.79:0/692054155 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5ab659c700 1 -- 172.21.15.79:0/692054155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ab80fcd00 msgr2=0x7f5ab810ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5ab659c700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ab80fcd00 0x7f5ab810ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.825+0000 7f5ab659c700 1 -- 172.21.15.79:0/692054155 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8005040 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.826+0000 7f5ab659c700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab810c4e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8004f90 tx=0x7f5aa8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.826+0000 7f5aa6ffd700 1 -- 172.21.15.79:0/692054155 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa800e040 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.826+0000 7f5aa6ffd700 1 -- 172.21.15.79:0/692054155 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5aa800bcd0 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.826+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab810db70 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.826+0000 7f5aa6ffd700 1 -- 172.21.15.79:0/692054155 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa801c630 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.826+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab810e0b0 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.829 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.827+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab804bb80 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.827+0000 7f5aa6ffd700 1 -- 172.21.15.79:0/692054155 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f5aa8013960 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.828+0000 7f5aa6ffd700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5aa0068ed0 0x7f5aa006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:06.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.828+0000 7f5aa6ffd700 1 -- 172.21.15.79:0/692054155 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f5aa8087460 con 0x7f5ab80fbaf0 2024-01-19T01:18:06.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.828+0000 7f5ab5d9b700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5aa0068ed0 0x7f5aa006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:06.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.828+0000 7f5ab5d9b700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5aa0068ed0 0x7f5aa006b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5aac006fd0 tx=0x7f5aac006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:18:06.833 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:06.832+0000 7f5aa6ffd700 1 -- 172.21.15.79:0/692054155 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5aa8058130 con 0x7f5ab80fbaf0 2024-01-19T01:18:07.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:07.034+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ab8101650 con 0x7f5aa0068ed0 2024-01-19T01:18:07.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:07 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:18:07.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:07 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:18:07.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:07 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:07.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:07 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:18:07.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:07 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:18:07.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:07 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:08.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:08 smithi079 ceph-mon[28374]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:18:08.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:08 smithi079 ceph-mon[28374]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:08.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:08 smithi152 ceph-mon[35694]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:18:08.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:08 smithi152 ceph-mon[35694]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:10.162 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: Detected new or changed devices on smithi079 2024-01-19T01:18:10.162 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.162 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.162 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: Adjusting osd_memory_target on smithi079 to 6432M 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:10.163 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:09 smithi079 ceph-mon[28374]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:10.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: Detected new or changed devices on smithi079 2024-01-19T01:18:10.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:10.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:10.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: Adjusting osd_memory_target on smithi079 to 6432M 2024-01-19T01:18:10.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:10.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:10.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:10.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:10.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:09 smithi152 ceph-mon[35694]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:10 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1761806555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d32cd7f-64af-4edf-b9cf-3db2519b9bd0"}]: dispatch 2024-01-19T01:18:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:10 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1761806555' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d32cd7f-64af-4edf-b9cf-3db2519b9bd0"}]': finished 2024-01-19T01:18:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:10 smithi079 ceph-mon[28374]: osdmap e16: 3 total, 2 up, 3 in 2024-01-19T01:18:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:10 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:11.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:10 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1761806555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d32cd7f-64af-4edf-b9cf-3db2519b9bd0"}]: dispatch 2024-01-19T01:18:11.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:10 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1761806555' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d32cd7f-64af-4edf-b9cf-3db2519b9bd0"}]': finished 2024-01-19T01:18:11.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:10 smithi152 ceph-mon[35694]: osdmap e16: 3 total, 2 up, 3 in 2024-01-19T01:18:11.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:10 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:11.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:11 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2268361653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:18:11.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:11 smithi079 ceph-mon[28374]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:12.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:11 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2268361653' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:18:12.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:11 smithi152 ceph-mon[35694]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:14.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:14 smithi079 ceph-mon[28374]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:14.884 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:14.885 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:14.885 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:14.885 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:14.885 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:14 smithi152 ceph-mon[35694]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:14.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:16.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:16 smithi079 ceph-mon[28374]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:16 smithi152 ceph-mon[35694]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:17.849 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-19T01:18:17.849 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-19T01:18:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:18.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:18 smithi152 ceph-mon[35694]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:18.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:18 smithi152 ceph-mon[35694]: Deploying daemon osd.2 on smithi079 2024-01-19T01:18:18.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:18 smithi079 ceph-mon[28374]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:18.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:18 smithi079 ceph-mon[28374]: Deploying daemon osd.2 on smithi079 2024-01-19T01:18:19.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:19.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:20.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:20 smithi152 ceph-mon[35694]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:20.933 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:20 smithi079 ceph-mon[28374]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:22 smithi152 ceph-mon[35694]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:22.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:22 smithi079 ceph-mon[28374]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:23.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:23.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:23.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:24.175 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:24.176 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:24.176 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:24.991 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:24 smithi079 ceph-mon[28374]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:25.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:24 smithi152 ceph-mon[35694]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:25.987 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 2 on host 'smithi079' 2024-01-19T01:18:25.987 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.985+0000 7f5aa6ffd700 1 -- 172.21.15.79:0/692054155 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5ab8101650 con 0x7f5aa0068ed0 2024-01-19T01:18:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5aa0068ed0 msgr2=0x7f5aa006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5aa0068ed0 0x7f5aa006b380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5aac006fd0 tx=0x7f5aac006db0 comp rx=0 tx=0).stop 2024-01-19T01:18:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 msgr2=0x7f5ab810c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab810c4e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8004f90 tx=0x7f5aa8009e30 comp rx=0 tx=0).stop 2024-01-19T01:18:25.989 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 shutdown_connections 2024-01-19T01:18:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5aa0068ed0 0x7f5aa006b380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5ab80fbaf0 0x7f5ab810c4e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 --2- 172.21.15.79:0/692054155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ab80fcd00 0x7f5ab810ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.988+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 >> 172.21.15.79:0/692054155 conn(0x7f5ab80f7190 msgr2=0x7f5ab80fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:18:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.989+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 shutdown_connections 2024-01-19T01:18:25.990 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:18:25.989+0000 7f5abcd7e700 1 -- 172.21.15.79:0/692054155 wait complete. 2024-01-19T01:18:26.862 DEBUG:teuthology.orchestra.run.smithi079:osd.2> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@osd.2.service 2024-01-19T01:18:26.864 INFO:tasks.cephadm:Deploying osd.3 on smithi152 with /dev/vg_nvme/lv_4... 2024-01-19T01:18:26.865 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-19T01:18:26.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:26 smithi079 ceph-mon[28374]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:26.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:26.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:26 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:26.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:26 smithi079 ceph-mon[28374]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-19T01:18:27.127 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:18:27.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:26 smithi152 ceph-mon[35694]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:27.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:27.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:26 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:27.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:26 smithi152 ceph-mon[35694]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-19T01:18:28.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:27 smithi152 ceph-mon[35694]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-19T01:18:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:27 smithi152 ceph-mon[35694]: osdmap e17: 3 total, 2 up, 3 in 2024-01-19T01:18:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:27 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:27 smithi152 ceph-mon[35694]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-01-19T01:18:28.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:27 smithi152 ceph-mon[35694]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:28.212 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:27 smithi079 ceph-mon[28374]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-19T01:18:28.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:27 smithi079 ceph-mon[28374]: osdmap e17: 3 total, 2 up, 3 in 2024-01-19T01:18:28.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:27 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:28.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:27 smithi079 ceph-mon[28374]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-01-19T01:18:28.213 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:27 smithi079 ceph-mon[28374]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:28.214 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:27 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-2[61535]: 2024-01-19T01:18:27.952+0000 7f6b3eb03700 -1 osd.2 0 waiting for initial osdmap 2024-01-19T01:18:28.214 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:27 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-2[61535]: 2024-01-19T01:18:27.957+0000 7f6b3a0ff700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T01:18:28.921 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-19T01:18:28.921 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-19T01:18:28.921 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-01-19T01:18:28.921 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-01-19T01:18:28.921 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113975 s, 920 MB/s 2024-01-19T01:18:28.921 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-01-19T01:18:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:28 smithi152 ceph-mon[35694]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-01-19T01:18:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:28 smithi152 ceph-mon[35694]: osdmap e18: 3 total, 2 up, 3 in 2024-01-19T01:18:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:28 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:28 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:29.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:28 smithi079 ceph-mon[28374]: from='osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-01-19T01:18:29.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:28 smithi079 ceph-mon[28374]: osdmap e18: 3 total, 2 up, 3 in 2024-01-19T01:18:29.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:28 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:29.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:28 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:29.503 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_4 2024-01-19T01:18:29.751 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: purged_snaps scrub starts 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: purged_snaps scrub ok 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] boot 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: osdmap e19: 3 total, 3 up, 3 in 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-01-19T01:18:30.152 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:30.153 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:29 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:30.164 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: purged_snaps scrub starts 2024-01-19T01:18:30.164 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: purged_snaps scrub ok 2024-01-19T01:18:30.164 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: osd.2 [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] boot 2024-01-19T01:18:30.165 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: osdmap e19: 3 total, 3 up, 3 in 2024-01-19T01:18:30.165 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-19T01:18:30.165 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-01-19T01:18:30.165 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-01-19T01:18:30.165 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:30.165 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:29 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:31.170 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-01-19T01:18:31.170 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:30 smithi079 ceph-mon[28374]: osdmap e20: 3 total, 3 up, 3 in 2024-01-19T01:18:31.170 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-19T01:18:31.221 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-01-19T01:18:31.221 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:30 smithi152 ceph-mon[35694]: osdmap e20: 3 total, 3 up, 3 in 2024-01-19T01:18:31.221 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-19T01:18:31.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/75556532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 msgr2=0x7f1df80fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:31.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/75556532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df80fe250 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1de8004940 tx=0x7f1de8009e30 comp rx=0 tx=0).stop 2024-01-19T01:18:31.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/75556532 shutdown_connections 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/75556532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df80fe250 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/75556532 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df80fcbf0 0x7f1df80fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/75556532 >> 172.21.15.152:0/75556532 conn(0x7f1df80f82d0 msgr2=0x7f1df80fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/75556532 shutdown_connections 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.245+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/75556532 wait complete. 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1dfd1cc700 1 Processor -- start 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1dfd1cc700 1 -- start start 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1dfd1cc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df80fcbf0 0x7f1df810d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1dfd1cc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df810db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1dfd1cc700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df810e120 con 0x7f1df80fcbf0 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1dfd1cc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df810e9c0 con 0x7f1df80fde00 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df659c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df810db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df659c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df810db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:57830/0 (socket says 172.21.15.152:57830) 2024-01-19T01:18:31.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df659c700 1 -- 172.21.15.152:0/369726693 learned_addr learned my addr 172.21.15.152:0/369726693 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df6d9d700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df80fcbf0 0x7f1df810d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df659c700 1 -- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df80fcbf0 msgr2=0x7f1df810d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df659c700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df80fcbf0 0x7f1df810d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df659c700 1 -- 172.21.15.152:0/369726693 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8005040 con 0x7f1df80fde00 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1df659c700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df810db20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1de800b3b0 tx=0x7f1de8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1de800e040 con 0x7f1df80fde00 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.246+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1de8013420 con 0x7f1df80fde00 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.247+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df810ec40 con 0x7f1df80fde00 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.247+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1de801c650 con 0x7f1df80fde00 2024-01-19T01:18:31.249 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.247+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df810f180 con 0x7f1df80fde00 2024-01-19T01:18:31.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.247+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df804bb80 con 0x7f1df80fde00 2024-01-19T01:18:31.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.248+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f1de8013590 con 0x7f1df80fde00 2024-01-19T01:18:31.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.248+0000 7f1ddf7fe700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1de0068ed0 0x7f1de006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:31.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.248+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f1de8087600 con 0x7f1df80fde00 2024-01-19T01:18:31.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.248+0000 7f1df6d9d700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1de0068ed0 0x7f1de006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:31.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.249+0000 7f1df6d9d700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1de0068ed0 0x7f1de006b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1df80fdc60 tx=0x7f1dec009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:18:31.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.253+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1de8058050 con 0x7f1df80fde00 2024-01-19T01:18:31.445 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.444+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1df810f640 con 0x7f1de0068ed0 2024-01-19T01:18:31.980 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:31.979+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1de8054990 con 0x7f1df80fde00 2024-01-19T01:18:31.992 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-19T01:18:31.992 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:31 smithi079 ceph-mon[28374]: osdmap e21: 3 total, 3 up, 3 in 2024-01-19T01:18:31.992 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:31 smithi079 ceph-mon[28374]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:31.992 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:31 smithi079 ceph-mon[28374]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:18:31.992 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:18:31.993 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:18:31.993 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:31 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:31.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:31 smithi079 sudo[63413]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T01:18:31.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:31 smithi079 sudo[63413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:31.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:31 smithi079 sudo[63413]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:31.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:31 smithi079 sudo[63413]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:32.150 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:31 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-19T01:18:32.150 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:31 smithi152 ceph-mon[35694]: osdmap e21: 3 total, 3 up, 3 in 2024-01-19T01:18:32.150 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:31 smithi152 ceph-mon[35694]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:32.150 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:31 smithi152 ceph-mon[35694]: from='client.24125 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:18:32.150 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:31 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:18:32.150 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:31 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:18:32.150 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:31 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:32.248 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66064]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T01:18:32.248 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66064]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:32.248 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66064]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:32.248 INFO:journalctl@ceph.osd.0.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66064]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:32.540 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66077]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T01:18:32.540 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66077]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:32.540 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66077]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:32.540 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66077]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:32.793 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66084]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T01:18:32.793 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66084]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:32.793 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66084]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:32.793 INFO:journalctl@ceph.osd.1.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66084]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:33.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66120]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T01:18:33.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66120]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:33.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66120]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:33.055 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:32 smithi079 sudo[66120]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: osdmap e22: 3 total, 3 up, 3 in 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: mgrmap e19: smithi079.miqczk(active, since 2m), standbys: smithi152.flyowb 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: Detected new or changed devices on smithi079 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: Adjusting osd_memory_target on smithi079 to 4288M 2024-01-19T01:18:33.056 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.057 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:33.057 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:33.057 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.057 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:33.176 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: osdmap e22: 3 total, 3 up, 3 in 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: mgrmap e19: smithi079.miqczk(active, since 2m), standbys: smithi152.flyowb 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: Detected new or changed devices on smithi079 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: Adjusting osd_memory_target on smithi079 to 4288M 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:33.177 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:33.178 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:33.317 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66189]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T01:18:33.318 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66189]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:33.318 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66189]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:33.318 INFO:journalctl@ceph.osd.2.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66189]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:33.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66276]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T01:18:33.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66276]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:33.581 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66276]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:33.582 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66276]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:33.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66296]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T01:18:33.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66296]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:33.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66296]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:33.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:33 smithi079 sudo[66296]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:33 smithi152 sudo[39161]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:33 smithi152 sudo[39161]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:33 smithi152 sudo[39161]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:33 smithi152 sudo[39161]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 sudo[39164]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 sudo[39164]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 sudo[39164]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-19T01:18:34.079 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 sudo[39164]: pam_unix(sudo:session): session closed for user root 2024-01-19T01:18:34.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:34 smithi079 ceph-mon[28374]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T01:18:34.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:34 smithi079 ceph-mon[28374]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:34.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:34 smithi079 ceph-mon[28374]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T01:18:34.346 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:18:34.347 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:18:34.347 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:18:34.347 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:18:34.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 ceph-mon[35694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T01:18:34.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 ceph-mon[35694]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:34.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 ceph-mon[35694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T01:18:34.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:18:34.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:18:34.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi079"}]: dispatch 2024-01-19T01:18:34.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mon metadata", "id": "smithi152"}]: dispatch 2024-01-19T01:18:35.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/2706515896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ac48076-9143-445e-bac1-0ae801b0d3fd"}]: dispatch 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ac48076-9143-445e-bac1-0ae801b0d3fd"}]: dispatch 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ac48076-9143-445e-bac1-0ae801b0d3fd"}]': finished 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: osdmap e23: 4 total, 3 up, 4 in 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:35.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:35 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/1605720060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:18:35.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/2706515896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ac48076-9143-445e-bac1-0ae801b0d3fd"}]: dispatch 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4ac48076-9143-445e-bac1-0ae801b0d3fd"}]: dispatch 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4ac48076-9143-445e-bac1-0ae801b0d3fd"}]': finished 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: osdmap e23: 4 total, 3 up, 4 in 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:35.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:35 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/1605720060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:18:36.359 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:36 smithi079 ceph-mon[28374]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:36.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:36 smithi152 ceph-mon[35694]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:38.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:37 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:38.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:37 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:38.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:37 smithi079 ceph-mon[28374]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:38.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:37 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:38.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:37 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:38.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:37 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:38.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:37 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:38.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:37 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:38.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:37 smithi152 ceph-mon[35694]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:38.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:37 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:38.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:37 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:38.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:37 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:40.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:40 smithi152 ceph-mon[35694]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-01-19T01:18:40.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:40 smithi079 ceph-mon[28374]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-01-19T01:18:41.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:41 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-19T01:18:41.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:41 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:41.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:41 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-19T01:18:41.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:41 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:42.764 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:42 smithi152 ceph-mon[35694]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-01-19T01:18:42.765 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:42 smithi152 ceph-mon[35694]: Deploying daemon osd.3 on smithi152 2024-01-19T01:18:42.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:42 smithi079 ceph-mon[28374]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-01-19T01:18:42.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:42 smithi079 ceph-mon[28374]: Deploying daemon osd.3 on smithi152 2024-01-19T01:18:44.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:44 smithi152 ceph-mon[35694]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-01-19T01:18:44.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:44 smithi079 ceph-mon[28374]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-01-19T01:18:46.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:46 smithi152 ceph-mon[35694]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-01-19T01:18:46.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:46 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:46.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:46 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:46.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:46 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:46.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:46 smithi079 ceph-mon[28374]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-01-19T01:18:46.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:46 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:46.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:46 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:46.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:46 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:48.699 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:48 smithi152 ceph-mon[35694]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T01:18:48.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:48 smithi079 ceph-mon[28374]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T01:18:49.678 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:49.679 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:49.679 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:49.679 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:49.679 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:49.679 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:18:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:49.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:50.418 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.417+0000 7f1ddf7fe700 1 -- 172.21.15.152:0/369726693 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1df810f640 con 0x7f1de0068ed0 2024-01-19T01:18:50.419 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 3 on host 'smithi152' 2024-01-19T01:18:50.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1de0068ed0 msgr2=0x7f1de006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:50.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1de0068ed0 0x7f1de006b380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1df80fdc60 tx=0x7f1dec009040 comp rx=0 tx=0).stop 2024-01-19T01:18:50.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 msgr2=0x7f1df810db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:50.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df810db20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1de800b3b0 tx=0x7f1de8009e30 comp rx=0 tx=0).stop 2024-01-19T01:18:50.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 shutdown_connections 2024-01-19T01:18:50.420 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1de0068ed0 0x7f1de006b380 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:50.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1df80fcbf0 0x7f1df810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:50.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 --2- 172.21.15.152:0/369726693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1df80fde00 0x7f1df810db20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:50.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 >> 172.21.15.152:0/369726693 conn(0x7f1df80f82d0 msgr2=0x7f1df8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:18:50.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 shutdown_connections 2024-01-19T01:18:50.421 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:50.419+0000 7f1dfd1cc700 1 -- 172.21.15.152:0/369726693 wait complete. 2024-01-19T01:18:50.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:50 smithi152 ceph-mon[35694]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T01:18:50.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:50 smithi152 ceph-mon[35694]: from='osd.3 [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T01:18:50.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:50 smithi152 ceph-mon[35694]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T01:18:50.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:50 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:50.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:50 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:50.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:50 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:50.960 DEBUG:teuthology.orchestra.run.smithi152:osd.3> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@osd.3.service 2024-01-19T01:18:50.965 INFO:tasks.cephadm:Deploying osd.4 on smithi152 with /dev/vg_nvme/lv_3... 2024-01-19T01:18:50.965 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-19T01:18:50.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:50 smithi079 ceph-mon[28374]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-01-19T01:18:50.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:50 smithi079 ceph-mon[28374]: from='osd.3 [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T01:18:50.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:50 smithi079 ceph-mon[28374]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-19T01:18:50.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:50 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:50.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:50 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:50.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:50 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:18:51.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:51 smithi152 ceph-mon[35694]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-19T01:18:51.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:51 smithi152 ceph-mon[35694]: osdmap e24: 4 total, 3 up, 4 in 2024-01-19T01:18:51.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:51 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:51.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:51 smithi152 ceph-mon[35694]: from='osd.3 [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:18:51.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:51 smithi152 ceph-mon[35694]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:18:51.674 INFO:journalctl@ceph.osd.3.smithi152.stdout:Jan 19 01:18:51 smithi152 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-3[41239]: 2024-01-19T01:18:51.545+0000 7fecb8359700 -1 osd.3 0 waiting for initial osdmap 2024-01-19T01:18:51.674 INFO:journalctl@ceph.osd.3.smithi152.stdout:Jan 19 01:18:51 smithi152 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-3[41239]: 2024-01-19T01:18:51.550+0000 7fecb1951700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T01:18:51.708 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:18:51.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:51 smithi079 ceph-mon[28374]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-19T01:18:51.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:51 smithi079 ceph-mon[28374]: osdmap e24: 4 total, 3 up, 4 in 2024-01-19T01:18:51.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:51 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:51.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:51 smithi079 ceph-mon[28374]: from='osd.3 [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:18:51.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:51 smithi079 ceph-mon[28374]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:18:52.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:52 smithi152 ceph-mon[35694]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:52.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:52 smithi152 ceph-mon[35694]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-01-19T01:18:52.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:52 smithi152 ceph-mon[35694]: osdmap e25: 4 total, 3 up, 4 in 2024-01-19T01:18:52.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:52 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:52.879 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:52 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:52.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:52 smithi079 ceph-mon[28374]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:52.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:52 smithi079 ceph-mon[28374]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-01-19T01:18:52.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:52 smithi079 ceph-mon[28374]: osdmap e25: 4 total, 3 up, 4 in 2024-01-19T01:18:52.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:52 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:52.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:52 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:53.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:53 smithi152 ceph-mon[35694]: purged_snaps scrub starts 2024-01-19T01:18:53.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:53 smithi152 ceph-mon[35694]: purged_snaps scrub ok 2024-01-19T01:18:53.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:53 smithi152 ceph-mon[35694]: osd.3 [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] boot 2024-01-19T01:18:53.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:53 smithi152 ceph-mon[35694]: osdmap e26: 4 total, 4 up, 4 in 2024-01-19T01:18:53.822 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:53 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:53.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:53 smithi079 ceph-mon[28374]: purged_snaps scrub starts 2024-01-19T01:18:53.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:53 smithi079 ceph-mon[28374]: purged_snaps scrub ok 2024-01-19T01:18:53.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:53 smithi079 ceph-mon[28374]: osd.3 [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] boot 2024-01-19T01:18:53.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:53 smithi079 ceph-mon[28374]: osdmap e26: 4 total, 4 up, 4 in 2024-01-19T01:18:53.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:53 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-19T01:18:54.406 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-19T01:18:54.406 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-19T01:18:54.406 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-01-19T01:18:54.406 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-01-19T01:18:54.406 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012785 s, 820 MB/s 2024-01-19T01:18:54.406 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 2024-01-19T01:18:54.407 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-01-19T01:18:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:54 smithi152 ceph-mon[35694]: pgmap v91: 1 pgs: 1 remapped+peering; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:54 smithi152 ceph-mon[35694]: osdmap e27: 4 total, 4 up, 4 in 2024-01-19T01:18:54.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:54 smithi079 ceph-mon[28374]: pgmap v91: 1 pgs: 1 remapped+peering; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-01-19T01:18:54.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:54 smithi079 ceph-mon[28374]: osdmap e27: 4 total, 4 up, 4 in 2024-01-19T01:18:55.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: osdmap e28: 4 total, 4 up, 4 in 2024-01-19T01:18:55.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:55.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:55.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:55.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:55.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:55.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:55.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:55 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:55.698 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_3 2024-01-19T01:18:55.946 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:18:55.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: osdmap e28: 4 total, 4 up, 4 in 2024-01-19T01:18:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:18:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:18:55.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:55 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:18:56.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:56 smithi152 ceph-mon[35694]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:18:56.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:56 smithi152 ceph-mon[35694]: Detected new or changed devices on smithi152 2024-01-19T01:18:56.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:56 smithi152 ceph-mon[35694]: Adjusting osd_memory_target on smithi152 to 15936M 2024-01-19T01:18:56.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:56 smithi152 ceph-mon[35694]: osdmap e29: 4 total, 4 up, 4 in 2024-01-19T01:18:56.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:56 smithi079 ceph-mon[28374]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:18:56.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:56 smithi079 ceph-mon[28374]: Detected new or changed devices on smithi152 2024-01-19T01:18:56.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:56 smithi079 ceph-mon[28374]: Adjusting osd_memory_target on smithi152 to 15936M 2024-01-19T01:18:56.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:56 smithi079 ceph-mon[28374]: osdmap e29: 4 total, 4 up, 4 in 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 -- 172.21.15.152:0/2251483950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540faed0 msgr2=0x7f84540fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 --2- 172.21.15.152:0/2251483950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540faed0 0x7f84540fb2e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8444004940 tx=0x7f8444009e30 comp rx=0 tx=0).stop 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 -- 172.21.15.152:0/2251483950 shutdown_connections 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 --2- 172.21.15.152:0/2251483950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84540fdf60 0x7f84540fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 --2- 172.21.15.152:0/2251483950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540faed0 0x7f84540fb2e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 -- 172.21.15.152:0/2251483950 >> 172.21.15.152:0/2251483950 conn(0x7f84540f6950 msgr2=0x7f84540f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 -- 172.21.15.152:0/2251483950 shutdown_connections 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.503+0000 7f845b632700 1 -- 172.21.15.152:0/2251483950 wait complete. 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f845b632700 1 Processor -- start 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f845b632700 1 -- start start 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f845b632700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84540faed0 0x7f84540fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f845b632700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540fdf60 0x7f84540fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:57.506 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f845b632700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540fc470 con 0x7f84540faed0 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f845b632700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540fc5e0 con 0x7f84540fdf60 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f8458bcd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540fdf60 0x7f84540fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f8458bcd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540fdf60 0x7f84540fbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:44798/0 (socket says 172.21.15.152:44798) 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f8458bcd700 1 -- 172.21.15.152:0/2458949400 learned_addr learned my addr 172.21.15.152:0/2458949400 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f84593ce700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84540faed0 0x7f84540fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f8458bcd700 1 -- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84540faed0 msgr2=0x7f84540fd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f8458bcd700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84540faed0 0x7f84540fd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.504+0000 7f8458bcd700 1 -- 172.21.15.152:0/2458949400 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8444005040 con 0x7f84540fdf60 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.505+0000 7f8458bcd700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540fdf60 0x7f84540fbe70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f845000c8c0 tx=0x7f84500044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.505+0000 7f8449ffb700 1 -- 172.21.15.152:0/2458949400 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f845000a760 con 0x7f84540fdf60 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.505+0000 7f8449ffb700 1 -- 172.21.15.152:0/2458949400 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f845000ad70 con 0x7f84540fdf60 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.505+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84540fc8c0 con 0x7f84540fdf60 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.505+0000 7f8449ffb700 1 -- 172.21.15.152:0/2458949400 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84500113a0 con 0x7f84540fdf60 2024-01-19T01:18:57.507 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.505+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8454115610 con 0x7f84540fdf60 2024-01-19T01:18:57.508 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.506+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84541075f0 con 0x7f84540fdf60 2024-01-19T01:18:57.508 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.506+0000 7f8449ffb700 1 -- 172.21.15.152:0/2458949400 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8450012400 con 0x7f84540fdf60 2024-01-19T01:18:57.508 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.507+0000 7f8449ffb700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8440068f60 0x7f844006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:18:57.508 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.507+0000 7f8449ffb700 1 -- 172.21.15.152:0/2458949400 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f8450086670 con 0x7f84540fdf60 2024-01-19T01:18:57.509 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.507+0000 7f84593ce700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8440068f60 0x7f844006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:18:57.509 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.508+0000 7f84593ce700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8440068f60 0x7f844006b410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8444004890 tx=0x7f8444009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:18:57.512 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.511+0000 7f8449ffb700 1 -- 172.21.15.152:0/2458949400 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8450056d10 con 0x7f84540fdf60 2024-01-19T01:18:57.703 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:18:57.703+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f845402b180 con 0x7f8440068f60 2024-01-19T01:18:57.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:57 smithi152 ceph-mon[35694]: osdmap e30: 4 total, 4 up, 4 in 2024-01-19T01:18:57.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:57 smithi079 ceph-mon[28374]: osdmap e30: 4 total, 4 up, 4 in 2024-01-19T01:18:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:58 smithi152 ceph-mon[35694]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:18:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:58 smithi152 ceph-mon[35694]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:18:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:58 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:18:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:58 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:18:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:18:58 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:18:58.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:58 smithi079 ceph-mon[28374]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:18:58.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:58 smithi079 ceph-mon[28374]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:18:58.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:58 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:18:58.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:58 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:18:58.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:18:58 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:00.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:00 smithi152 ceph-mon[35694]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-01-19T01:19:00.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:00 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/696172194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c505c3b-0853-4af4-bb1b-4e32ba16bede"}]: dispatch 2024-01-19T01:19:00.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:00 smithi152 ceph-mon[35694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c505c3b-0853-4af4-bb1b-4e32ba16bede"}]: dispatch 2024-01-19T01:19:00.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:00 smithi152 ceph-mon[35694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c505c3b-0853-4af4-bb1b-4e32ba16bede"}]': finished 2024-01-19T01:19:00.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:00 smithi152 ceph-mon[35694]: osdmap e31: 5 total, 4 up, 5 in 2024-01-19T01:19:00.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:00 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:00.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:00 smithi079 ceph-mon[28374]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-01-19T01:19:00.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:00 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/696172194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c505c3b-0853-4af4-bb1b-4e32ba16bede"}]: dispatch 2024-01-19T01:19:00.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:00 smithi079 ceph-mon[28374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c505c3b-0853-4af4-bb1b-4e32ba16bede"}]: dispatch 2024-01-19T01:19:00.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:00 smithi079 ceph-mon[28374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c505c3b-0853-4af4-bb1b-4e32ba16bede"}]': finished 2024-01-19T01:19:00.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:00 smithi079 ceph-mon[28374]: osdmap e31: 5 total, 4 up, 5 in 2024-01-19T01:19:00.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:00 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:01.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:01 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/255673930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:19:01.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:01 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/255673930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:19:02.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:02 smithi152 ceph-mon[35694]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-19T01:19:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:02 smithi079 ceph-mon[28374]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-19T01:19:04.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:04 smithi152 ceph-mon[35694]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-01-19T01:19:04.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:04 smithi079 ceph-mon[28374]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-01-19T01:19:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:06.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:06 smithi152 ceph-mon[35694]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-01-19T01:19:06.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:06 smithi079 ceph-mon[28374]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-01-19T01:19:07.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:07 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-19T01:19:07.869 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:07 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:07.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:07 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-19T01:19:07.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:07 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:08.835 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:08 smithi152 ceph-mon[35694]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-19T01:19:08.835 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:08 smithi152 ceph-mon[35694]: Deploying daemon osd.4 on smithi152 2024-01-19T01:19:08.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:08 smithi079 ceph-mon[28374]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-19T01:19:08.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:08 smithi079 ceph-mon[28374]: Deploying daemon osd.4 on smithi152 2024-01-19T01:19:10.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:10 smithi152 ceph-mon[35694]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:10 smithi079 ceph-mon[28374]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:12.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:12 smithi152 ceph-mon[35694]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:12.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:12.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:12.863 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:12 smithi079 ceph-mon[28374]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:12.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:12.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:14 smithi152 ceph-mon[35694]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:14.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:14 smithi079 ceph-mon[28374]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:15.517 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.516+0000 7f8449ffb700 1 -- 172.21.15.152:0/2458949400 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f845402b180 con 0x7f8440068f60 2024-01-19T01:19:15.518 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 4 on host 'smithi152' 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8440068f60 msgr2=0x7f844006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8440068f60 0x7f844006b410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8444004890 tx=0x7f8444009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540fdf60 msgr2=0x7f84540fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540fdf60 0x7f84540fbe70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f845000c8c0 tx=0x7f84500044c0 comp rx=0 tx=0).stop 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 shutdown_connections 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8440068f60 0x7f844006b410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f84540faed0 0x7f84540fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 --2- 172.21.15.152:0/2458949400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f84540fdf60 0x7f84540fbe70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 >> 172.21.15.152:0/2458949400 conn(0x7f84540f6950 msgr2=0x7f84540f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 shutdown_connections 2024-01-19T01:19:15.520 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:15.519+0000 7f845b632700 1 -- 172.21.15.152:0/2458949400 wait complete. 2024-01-19T01:19:16.239 DEBUG:teuthology.orchestra.run.smithi152:osd.4> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@osd.4.service 2024-01-19T01:19:16.241 INFO:tasks.cephadm:Deploying osd.5 on smithi152 with /dev/vg_nvme/lv_2... 2024-01-19T01:19:16.241 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-19T01:19:16.624 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:19:16.748 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:16 smithi152 ceph-mon[35694]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:16.748 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:16.748 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:16.748 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:16 smithi152 ceph-mon[35694]: from='osd.4 [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T01:19:16.748 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:16 smithi152 ceph-mon[35694]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T01:19:16.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:16 smithi079 ceph-mon[28374]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:16.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:16.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:16.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:16 smithi079 ceph-mon[28374]: from='osd.4 [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T01:19:16.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:16 smithi079 ceph-mon[28374]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-19T01:19:17.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-mon[35694]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-19T01:19:17.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-mon[35694]: osdmap e32: 5 total, 4 up, 5 in 2024-01-19T01:19:17.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:17.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-mon[35694]: from='osd.4 [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:17.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-mon[35694]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:17.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:17.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:17.674 INFO:journalctl@ceph.osd.4.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-4[47656]: 2024-01-19T01:19:17.621+0000 7f66089da700 -1 osd.4 0 waiting for initial osdmap 2024-01-19T01:19:17.676 INFO:journalctl@ceph.osd.4.smithi152.stdout:Jan 19 01:19:17 smithi152 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-4[47656]: 2024-01-19T01:19:17.628+0000 7f6601fd2700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T01:19:17.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:17 smithi079 ceph-mon[28374]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-19T01:19:17.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:17 smithi079 ceph-mon[28374]: osdmap e32: 5 total, 4 up, 5 in 2024-01-19T01:19:17.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:17.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:17 smithi079 ceph-mon[28374]: from='osd.4 [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:17.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:17 smithi079 ceph-mon[28374]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:17.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:17.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:18.521 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-19T01:19:18.521 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-19T01:19:18.522 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-01-19T01:19:18.522 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-01-19T01:19:18.522 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127192 s, 824 MB/s 2024-01-19T01:19:18.522 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-01-19T01:19:18.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:18 smithi152 ceph-mon[35694]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:18.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:18 smithi152 ceph-mon[35694]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-01-19T01:19:18.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:18 smithi152 ceph-mon[35694]: osdmap e33: 5 total, 4 up, 5 in 2024-01-19T01:19:18.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:18 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:18.877 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:18 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:18.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:18 smithi079 ceph-mon[28374]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-01-19T01:19:18.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:18 smithi079 ceph-mon[28374]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-01-19T01:19:18.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:18 smithi079 ceph-mon[28374]: osdmap e33: 5 total, 4 up, 5 in 2024-01-19T01:19:18.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:18 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:18.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:18 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:19.906 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:19 smithi152 ceph-mon[35694]: purged_snaps scrub starts 2024-01-19T01:19:19.906 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:19 smithi152 ceph-mon[35694]: purged_snaps scrub ok 2024-01-19T01:19:19.906 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:19 smithi152 ceph-mon[35694]: osd.4 [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] boot 2024-01-19T01:19:19.906 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:19 smithi152 ceph-mon[35694]: osdmap e34: 5 total, 5 up, 5 in 2024-01-19T01:19:19.906 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:19.906 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:19.929 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_2 2024-01-19T01:19:19.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:19 smithi079 ceph-mon[28374]: purged_snaps scrub starts 2024-01-19T01:19:19.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:19 smithi079 ceph-mon[28374]: purged_snaps scrub ok 2024-01-19T01:19:19.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:19 smithi079 ceph-mon[28374]: osd.4 [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] boot 2024-01-19T01:19:19.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:19 smithi079 ceph-mon[28374]: osdmap e34: 5 total, 5 up, 5 in 2024-01-19T01:19:19.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-19T01:19:19.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:20.416 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:19:20.839 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:20 smithi152 ceph-mon[35694]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:20.840 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:20 smithi152 ceph-mon[35694]: osdmap e35: 5 total, 5 up, 5 in 2024-01-19T01:19:20.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:20 smithi079 ceph-mon[28374]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:20.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:20 smithi079 ceph-mon[28374]: osdmap e35: 5 total, 5 up, 5 in 2024-01-19T01:19:22.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- 172.21.15.152:0/960424300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 msgr2=0x7fe5c00fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:22.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/960424300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c00fbf00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4004940 tx=0x7fe5b4009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- 172.21.15.152:0/960424300 shutdown_connections 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/960424300 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5c00fcd00 0x7fe5c00fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/960424300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c00fbf00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- 172.21.15.152:0/960424300 >> 172.21.15.152:0/960424300 conn(0x7fe5c00f71d0 msgr2=0x7fe5c00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- 172.21.15.152:0/960424300 shutdown_connections 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- 172.21.15.152:0/960424300 wait complete. 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 Processor -- start 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- start start 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c010c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5c00fcd00 0x7fe5c010c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c010d040 con 0x7fe5c00fcd00 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.525+0000 7fe5c6b76700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c010d880 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c4912700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c010c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c4912700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c010c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:46188/0 (socket says 172.21.15.152:46188) 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c4912700 1 -- 172.21.15.152:0/2435737396 learned_addr learned my addr 172.21.15.152:0/2435737396 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5bffff700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5c00fcd00 0x7fe5c010c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c4912700 1 -- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5c00fcd00 msgr2=0x7fe5c010c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c4912700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5c00fcd00 0x7fe5c010c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c4912700 1 -- 172.21.15.152:0/2435737396 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b4005040 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5bffff700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5c00fcd00 0x7fe5c010c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c4912700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c010c470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4009360 tx=0x7fe5b4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5bd7fa700 1 -- 172.21.15.152:0/2435737396 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5b400e040 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c010db00 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5bd7fa700 1 -- 172.21.15.152:0/2435737396 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5b4004290 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5bd7fa700 1 -- 172.21.15.152:0/2435737396 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5b401c630 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.526+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c010e040 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.527+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c004bb80 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.528+0000 7fe5bd7fa700 1 -- 172.21.15.152:0/2435737396 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe5b4013420 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.528+0000 7fe5bd7fa700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe5b0068f70 0x7fe5b006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:22.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.528+0000 7fe5bd7fa700 1 -- 172.21.15.152:0/2435737396 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fe5b4087c70 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.530 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.528+0000 7fe5bffff700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe5b0068f70 0x7fe5b006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:22.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.529+0000 7fe5bffff700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe5b0068f70 0x7fe5b006b420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac006fd0 tx=0x7fe5ac008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:22.533 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.532+0000 7fe5bd7fa700 1 -- 172.21.15.152:0/2435737396 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe5b40580c0 con 0x7fe5c00fbaf0 2024-01-19T01:19:22.729 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:22.728+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5c010dc90 con 0x7fe5b0068f70 2024-01-19T01:19:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: Detected new or changed devices on smithi152 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: Adjusting osd_memory_target on smithi152 to 7968M 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:22.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:22.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:22.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:22.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:22.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: Detected new or changed devices on smithi152 2024-01-19T01:19:22.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:22.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:22.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:22.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:22.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: Adjusting osd_memory_target on smithi152 to 7968M 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:23.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:23.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:23 smithi152 ceph-mon[35694]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:19:23.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:19:23.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:19:23.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:23 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:23.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:23 smithi079 ceph-mon[28374]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:19:23.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-19T01:19:23.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-19T01:19:23.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:23 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:24.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:24 smithi152 ceph-mon[35694]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:24.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:24 smithi079 ceph-mon[28374]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:25.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:25 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/523613767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c343c013-2f74-4c20-83bd-14e41c304d62"}]: dispatch 2024-01-19T01:19:25.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:25 smithi152 ceph-mon[35694]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c343c013-2f74-4c20-83bd-14e41c304d62"}]: dispatch 2024-01-19T01:19:25.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:25 smithi152 ceph-mon[35694]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c343c013-2f74-4c20-83bd-14e41c304d62"}]': finished 2024-01-19T01:19:25.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:25 smithi152 ceph-mon[35694]: osdmap e36: 6 total, 5 up, 6 in 2024-01-19T01:19:25.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:25 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:25.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:25 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/523613767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c343c013-2f74-4c20-83bd-14e41c304d62"}]: dispatch 2024-01-19T01:19:25.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:25 smithi079 ceph-mon[28374]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c343c013-2f74-4c20-83bd-14e41c304d62"}]: dispatch 2024-01-19T01:19:25.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:25 smithi079 ceph-mon[28374]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c343c013-2f74-4c20-83bd-14e41c304d62"}]': finished 2024-01-19T01:19:25.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:25 smithi079 ceph-mon[28374]: osdmap e36: 6 total, 5 up, 6 in 2024-01-19T01:19:25.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:25 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:26.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:26 smithi152 ceph-mon[35694]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:26.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:26 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/2532327962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:19:26.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:26 smithi079 ceph-mon[28374]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:26.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:26 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/2532327962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-19T01:19:28.909 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:28 smithi152 ceph-mon[35694]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:28.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:28 smithi079 ceph-mon[28374]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:30.881 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:30 smithi152 ceph-mon[35694]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:30.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:30 smithi079 ceph-mon[28374]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:32.816 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:32 smithi152 ceph-mon[35694]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:32.816 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-19T01:19:32.816 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:32 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:32.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:32 smithi079 ceph-mon[28374]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:32.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-19T01:19:32.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:32 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:33.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:33 smithi152 ceph-mon[35694]: Deploying daemon osd.5 on smithi152 2024-01-19T01:19:33.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:33 smithi079 ceph-mon[28374]: Deploying daemon osd.5 on smithi152 2024-01-19T01:19:34.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:34 smithi152 ceph-mon[35694]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:34.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:34 smithi079 ceph-mon[28374]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:36.719 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:36 smithi152 ceph-mon[35694]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:36.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:36 smithi079 ceph-mon[28374]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:38.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:38 smithi152 ceph-mon[35694]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:38.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:38.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:38.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:38 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:38.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:38 smithi079 ceph-mon[28374]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:38.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:38.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:38.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:38 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:40.444 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.443+0000 7fe5bd7fa700 1 -- 172.21.15.152:0/2435737396 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe5c010dc90 con 0x7fe5b0068f70 2024-01-19T01:19:40.444 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 5 on host 'smithi152' 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe5b0068f70 msgr2=0x7fe5b006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe5b0068f70 0x7fe5b006b420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac006fd0 tx=0x7fe5ac008040 comp rx=0 tx=0).stop 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 msgr2=0x7fe5c010c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c010c470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4009360 tx=0x7fe5b4009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 shutdown_connections 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe5b0068f70 0x7fe5b006b420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c00fbaf0 0x7fe5c010c470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 --2- 172.21.15.152:0/2435737396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe5c00fcd00 0x7fe5c010c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 >> 172.21.15.152:0/2435737396 conn(0x7fe5c00f71d0 msgr2=0x7fe5c00fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:40.447 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 shutdown_connections 2024-01-19T01:19:40.448 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:40.446+0000 7fe5c6b76700 1 -- 172.21.15.152:0/2435737396 wait complete. 2024-01-19T01:19:40.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:40 smithi152 ceph-mon[35694]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:40.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:40 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:40.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:40 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:40.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:40 smithi079 ceph-mon[28374]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:40.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:40.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:40 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:41.715 DEBUG:teuthology.orchestra.run.smithi152:osd.5> sudo journalctl -f -n 0 -u ceph-f3747e56-b667-11ee-95ae-87774f69a715@osd.5.service 2024-01-19T01:19:41.718 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-01-19T01:19:41.718 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd stat -f json 2024-01-19T01:19:41.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:41 smithi152 ceph-mon[35694]: from='osd.5 [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T01:19:41.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:41 smithi152 ceph-mon[35694]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T01:19:41.990 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:19:41.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:41 smithi079 ceph-mon[28374]: from='osd.5 [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T01:19:41.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:41 smithi079 ceph-mon[28374]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-19T01:19:42.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:42.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-19T01:19:42.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: osdmap e37: 6 total, 5 up, 6 in 2024-01-19T01:19:42.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:42.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: from='osd.5 [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:42.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:42.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:42.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:42 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: osdmap e37: 6 total, 5 up, 6 in 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: from='osd.5 [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:42.861 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:42.862 INFO:journalctl@ceph.osd.5.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-5[54105]: 2024-01-19T01:19:42.533+0000 7fabc8435700 -1 osd.5 0 waiting for initial osdmap 2024-01-19T01:19:42.862 INFO:journalctl@ceph.osd.5.smithi152.stdout:Jan 19 01:19:42 smithi152 ceph-f3747e56-b667-11ee-95ae-87774f69a715-osd-5[54105]: 2024-01-19T01:19:42.541+0000 7fabc2a2f700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-19T01:19:43.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 -- 172.21.15.79:0/2508552835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 msgr2=0x7f09240fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:43.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 --2- 172.21.15.79:0/2508552835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f09240fe250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0918004940 tx=0x7f0918009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:43.665 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 -- 172.21.15.79:0/2508552835 shutdown_connections 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 --2- 172.21.15.79:0/2508552835 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f09240fe250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 --2- 172.21.15.79:0/2508552835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09240fcbf0 0x7f09240fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 -- 172.21.15.79:0/2508552835 >> 172.21.15.79:0/2508552835 conn(0x7f09240f82d0 msgr2=0x7f09240fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 -- 172.21.15.79:0/2508552835 shutdown_connections 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.662+0000 7f092a13b700 1 -- 172.21.15.79:0/2508552835 wait complete. 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f092a13b700 1 Processor -- start 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f092a13b700 1 -- start start 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f092a13b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09240fcbf0 0x7f0924071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f092a13b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f092406ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f092a13b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0924070620 con 0x7f09240fde00 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f092a13b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0924070790 con 0x7f09240fcbf0 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f0922ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f092406ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f0922ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f092406ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50240/0 (socket says 172.21.15.79:50240) 2024-01-19T01:19:43.666 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f0922ffd700 1 -- 172.21.15.79:0/1861058860 learned_addr learned my addr 172.21.15.79:0/1861058860 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.663+0000 7f09237fe700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09240fcbf0 0x7f0924071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f0922ffd700 1 -- 172.21.15.79:0/1861058860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09240fcbf0 msgr2=0x7f0924071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f0922ffd700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09240fcbf0 0x7f0924071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f0922ffd700 1 -- 172.21.15.79:0/1861058860 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918005040 con 0x7f09240fde00 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f0922ffd700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f092406ff90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0918009fd0 tx=0x7f0918009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f090bfff700 1 -- 172.21.15.79:0/1861058860 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f091800e040 con 0x7f09240fde00 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0924070a10 con 0x7f09240fde00 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f090bfff700 1 -- 172.21.15.79:0/1861058860 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0918004290 con 0x7f09240fde00 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f090bfff700 1 -- 172.21.15.79:0/1861058860 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f091801c3f0 con 0x7f09240fde00 2024-01-19T01:19:43.667 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.664+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0924115590 con 0x7f09240fde00 2024-01-19T01:19:43.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.665+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092404bb80 con 0x7f09240fde00 2024-01-19T01:19:43.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.665+0000 7f090bfff700 1 -- 172.21.15.79:0/1861058860 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f091800bc50 con 0x7f09240fde00 2024-01-19T01:19:43.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.666+0000 7f090bfff700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f090c068f70 0x7f090c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:43.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.666+0000 7f090bfff700 1 -- 172.21.15.79:0/1861058860 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0918087e40 con 0x7f09240fde00 2024-01-19T01:19:43.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.666+0000 7f09237fe700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f090c068f70 0x7f090c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:43.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.666+0000 7f09237fe700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f090c068f70 0x7f090c06b420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09240718a0 tx=0x7f09140091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:43.671 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.670+0000 7f090bfff700 1 -- 172.21.15.79:0/1861058860 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f09180580e0 con 0x7f09240fde00 2024-01-19T01:19:43.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:43 smithi079 ceph-mon[28374]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-01-19T01:19:43.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:43 smithi079 ceph-mon[28374]: osdmap e38: 6 total, 5 up, 6 in 2024-01-19T01:19:43.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:43 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:43.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:43 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:43.818 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:43 smithi152 ceph-mon[35694]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-01-19T01:19:43.818 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:43 smithi152 ceph-mon[35694]: osdmap e38: 6 total, 5 up, 6 in 2024-01-19T01:19:43.818 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:43.818 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:43.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.857+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0924115910 con 0x7f09240fde00 2024-01-19T01:19:43.858 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.857+0000 7f090bfff700 1 -- 172.21.15.79:0/1861058860 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f0918057c70 con 0x7f09240fde00 2024-01-19T01:19:43.858 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:19:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f090c068f70 msgr2=0x7f090c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f090c068f70 0x7f090c06b420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f09240718a0 tx=0x7f09140091a0 comp rx=0 tx=0).stop 2024-01-19T01:19:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 msgr2=0x7f092406ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f092406ff90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0918009fd0 tx=0x7f0918009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 shutdown_connections 2024-01-19T01:19:43.860 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f090c068f70 0x7f090c06b420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09240fcbf0 0x7f0924071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 --2- 172.21.15.79:0/1861058860 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f09240fde00 0x7f092406ff90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 >> 172.21.15.79:0/1861058860 conn(0x7f09240f82d0 msgr2=0x7f0924101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 shutdown_connections 2024-01-19T01:19:43.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:43.859+0000 7f092a13b700 1 -- 172.21.15.79:0/1861058860 wait complete. 2024-01-19T01:19:44.623 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1705627183,"num_in_osds":6,"osd_in_since":1705627165,"num_remapped_pgs":0} 2024-01-19T01:19:44.624 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-19T01:19:44.885 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:19:44.897 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:44 smithi079 ceph-mon[28374]: purged_snaps scrub starts 2024-01-19T01:19:44.898 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:44 smithi079 ceph-mon[28374]: purged_snaps scrub ok 2024-01-19T01:19:44.898 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:44 smithi079 ceph-mon[28374]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:44.898 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:44 smithi079 ceph-mon[28374]: osd.5 [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] boot 2024-01-19T01:19:44.898 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:44 smithi079 ceph-mon[28374]: osdmap e39: 6 total, 6 up, 6 in 2024-01-19T01:19:44.898 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:44 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:44.898 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:44 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1861058860' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-19T01:19:44.904 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:44 smithi152 ceph-mon[35694]: purged_snaps scrub starts 2024-01-19T01:19:44.904 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:44 smithi152 ceph-mon[35694]: purged_snaps scrub ok 2024-01-19T01:19:44.904 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:44 smithi152 ceph-mon[35694]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-01-19T01:19:44.905 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:44 smithi152 ceph-mon[35694]: osd.5 [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] boot 2024-01-19T01:19:44.905 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:44 smithi152 ceph-mon[35694]: osdmap e39: 6 total, 6 up, 6 in 2024-01-19T01:19:44.905 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:44 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-19T01:19:44.905 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:44 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1861058860' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-19T01:19:45.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: osdmap e40: 6 total, 6 up, 6 in 2024-01-19T01:19:45.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:45.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:45.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:45.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:45.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:45.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.674 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:45 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:45.856 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: osdmap e40: 6 total, 6 up, 6 in 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:45.857 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:45 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 -- 172.21.15.79:0/787563321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 msgr2=0x7f1718101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 --2- 172.21.15.79:0/787563321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f1718101ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1708004940 tx=0x7f1708009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 -- 172.21.15.79:0/787563321 shutdown_connections 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 --2- 172.21.15.79:0/787563321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17180fbb40 0x7f17180fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 --2- 172.21.15.79:0/787563321 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f1718101ed0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 -- 172.21.15.79:0/787563321 >> 172.21.15.79:0/787563321 conn(0x7f17180f76c0 msgr2=0x7f17180f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 -- 172.21.15.79:0/787563321 shutdown_connections 2024-01-19T01:19:46.501 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.496+0000 7f171cc63700 1 -- 172.21.15.79:0/787563321 wait complete. 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f171cc63700 1 Processor -- start 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f171cc63700 1 -- start start 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f171cc63700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17180fbb40 0x7f171810e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f171cc63700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f171810ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f171cc63700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171810f260 con 0x7f1718101b00 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f171cc63700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171810f3d0 con 0x7f17180fbb40 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f1715d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f171810ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f1715d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f171810ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50250/0 (socket says 172.21.15.79:50250) 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f1715d9b700 1 -- 172.21.15.79:0/2437756378 learned_addr learned my addr 172.21.15.79:0/2437756378 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f171659c700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17180fbb40 0x7f171810e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f1715d9b700 1 -- 172.21.15.79:0/2437756378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17180fbb40 msgr2=0x7f171810e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f1715d9b700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17180fbb40 0x7f171810e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.497+0000 7f1715d9b700 1 -- 172.21.15.79:0/2437756378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1708005040 con 0x7f1718101b00 2024-01-19T01:19:46.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.498+0000 7f1715d9b700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f171810ec40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f170c00b7c0 tx=0x7f170c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.498+0000 7f1706ffd700 1 -- 172.21.15.79:0/2437756378 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f170c00f6f0 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.498+0000 7f1706ffd700 1 -- 172.21.15.79:0/2437756378 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f170c00fd00 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.498+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1718117720 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.498+0000 7f1706ffd700 1 -- 172.21.15.79:0/2437756378 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f170c011e90 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.498+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1718117c30 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.499+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1718103fb0 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.499+0000 7f1706ffd700 1 -- 172.21.15.79:0/2437756378 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f170c00f850 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.499+0000 7f1706ffd700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1700068f60 0x7f170006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.500+0000 7f171659c700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1700068f60 0x7f170006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.500+0000 7f1706ffd700 1 -- 172.21.15.79:0/2437756378 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f170c086900 con 0x7f1718101b00 2024-01-19T01:19:46.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.500+0000 7f171659c700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1700068f60 0x7f170006b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1708004560 tx=0x7f1708009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:46.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.504+0000 7f1706ffd700 1 -- 172.21.15.79:0/2437756378 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f170c053110 con 0x7f1718101b00 2024-01-19T01:19:46.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.691+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17180fb2c0 con 0x7f1718101b00 2024-01-19T01:19:46.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:46 smithi079 ceph-mon[28374]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:46.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:46 smithi079 ceph-mon[28374]: Detected new or changed devices on smithi152 2024-01-19T01:19:46.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:46 smithi079 ceph-mon[28374]: Adjusting osd_memory_target on smithi152 to 5312M 2024-01-19T01:19:46.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:46 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:46.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:46 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:46.693 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:46 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:46.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.691+0000 7f1706ffd700 1 -- 172.21.15.79:0/2437756378 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11091 (secure 0 0 0) 0x7f170c056730 con 0x7f1718101b00 2024-01-19T01:19:46.693 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:19:46.693 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","created":"2024-01-19T01:14:07.716096+0000","modified":"2024-01-19T01:19:44.537500+0000","last_up_change":"2024-01-19T01:19:43.531498+0000","last_in_change":"2024-01-19T01:19:25.050496+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T01:18:29.500877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4500f84c-0d7c-472b-8d18-35de388ec5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6803","nonce":2692943028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6805","nonce":2692943028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6809","nonce":2692943028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6807","nonce":2692943028}]},"public_addr":"172.21.15.79:6803/2692943028","cluster_addr":"172.21.15.79:6805/2692943028","heartbeat_back_addr":"172.21.15.79:6809/2692943028","heartbeat_front_addr":"172.21.15.79:6807/2692943028","state":["exists","up"]},{"osd":1,"uuid":"0286a58d-91d9-48f3-9a73-cad13c064623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6811","nonce":3254175144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6813","nonce":3254175144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6817","nonce":3254175144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6815","nonce":3254175144}]},"public_addr":"172.21.15.79:6811/3254175144","cluster_addr":"172.21.15.79:6813/3254175144","heartbeat_back_addr":"172.21.15.79:6817/3254175144","heartbeat_front_addr":"172.21.15.79:6815/3254175144","state":["exists","up"]},{"osd":2,"uuid":"1d32cd7f-64af-4edf-b9cf-3db2519b9bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6819","nonce":3125547025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6821","nonce":3125547025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6825","nonce":3125547025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6823","nonce":3125547025}]},"public_addr":"172.21.15.79:6819/3125547025","cluster_addr":"172.21.15.79:6821/3125547025","heartbeat_back_addr":"172.21.15.79:6825/3125547025","heartbeat_front_addr":"172.21.15.79:6823/3125547025","state":["exists","up"]},{"osd":3,"uuid":"4ac48076-9143-445e-bac1-0ae801b0d3fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6801","nonce":2401180651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6803","nonce":2401180651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6807","nonce":2401180651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6805","nonce":2401180651}]},"public_addr":"172.21.15.152:6801/2401180651","cluster_addr":"172.21.15.152:6803/2401180651","heartbeat_back_addr":"172.21.15.152:6807/2401180651","heartbeat_front_addr":"172.21.15.152:6805/2401180651","state":["exists","up"]},{"osd":4,"uuid":"7c505c3b-0853-4af4-bb1b-4e32ba16bede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6809","nonce":1289083445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6811","nonce":1289083445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6815","nonce":1289083445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6813","nonce":1289083445}]},"public_addr":"172.21.15.152:6809/1289083445","cluster_addr":"172.21.15.152:6811/1289083445","heartbeat_back_addr":"172.21.15.152:6815/1289083445","heartbeat_front_addr":"172.21.15.152:6813/1289083445","state":["exists","up"]},{"osd":5,"uuid":"c343c013-2f74-4c20-83bd-14e41c304d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6817","nonce":3123617721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6819","nonce":3123617721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6823","nonce":3123617721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6821","nonce":3123617721}]},"public_addr":"172.21.15.152:6817/3123617721","cluster_addr":"172.21.15.152:6819/3123617721","heartbeat_back_addr":"172.21.15.152:6823/3123617721","heartbeat_front_addr":"172.21.15.152:6821/3123617721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:17:31.789908+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:00.346643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:27.657969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:51.195401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:17.092535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:42.162295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1393070546":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6800/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/780174951":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/207141703":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/1174599613":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6801/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6800/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6801/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6801/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6800/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/2671943691":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/4252083415":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/907783696":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/2783373766":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/547589622":"2024-01-20T01:14:38.242827+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T01:19:46.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1700068f60 msgr2=0x7f170006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:46.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1700068f60 0x7f170006b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1708004560 tx=0x7f1708009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 msgr2=0x7f171810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f171810ec40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f170c00b7c0 tx=0x7f170c009250 comp rx=0 tx=0).stop 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 shutdown_connections 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1700068f60 0x7f170006b410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17180fbb40 0x7f171810e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 --2- 172.21.15.79:0/2437756378 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1718101b00 0x7f171810ec40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.693+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 >> 172.21.15.79:0/2437756378 conn(0x7f17180f76c0 msgr2=0x7f17180f91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.694+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 shutdown_connections 2024-01-19T01:19:46.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:46.694+0000 7f171cc63700 1 -- 172.21.15.79:0/2437756378 wait complete. 2024-01-19T01:19:46.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:46 smithi152 ceph-mon[35694]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:46.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:46 smithi152 ceph-mon[35694]: Detected new or changed devices on smithi152 2024-01-19T01:19:46.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:46 smithi152 ceph-mon[35694]: Adjusting osd_memory_target on smithi152 to 5312M 2024-01-19T01:19:46.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:46 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:19:46.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:46 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:19:46.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:46 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:19:47.475 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-19T01:18:29.500877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-01-19T01:19:47.476 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-01-19T01:19:47.732 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:19:47.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:47 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2437756378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:19:47.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:47 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2437756378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:19:48.824 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:48 smithi152 ceph-mon[35694]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:48.905 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:48 smithi079 ceph-mon[28374]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 -- 172.21.15.79:0/2044440157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb00fb1e0 msgr2=0x7f7fb00fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/2044440157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb00fb1e0 0x7f7fb00fb630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4004940 tx=0x7f7fa4009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 -- 172.21.15.79:0/2044440157 shutdown_connections 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/2044440157 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb00fb1e0 0x7f7fb00fb630 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/2044440157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb0102c70 0x7f7fb0103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 -- 172.21.15.79:0/2044440157 >> 172.21.15.79:0/2044440157 conn(0x7f7fb00711a0 msgr2=0x7f7fb00715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 -- 172.21.15.79:0/2044440157 shutdown_connections 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.441+0000 7f7fb54cd700 1 -- 172.21.15.79:0/2044440157 wait complete. 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fb54cd700 1 Processor -- start 2024-01-19T01:19:49.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fb54cd700 1 -- start start 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fb54cd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb00fb1e0 0x7f7fb0113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fb54cd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb0102c70 0x7f7fb0113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fb54cd700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb01141e0 con 0x7f7fb0102c70 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fb54cd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb010d4d0 con 0x7f7fb00fb1e0 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fae7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb0102c70 0x7f7fb0113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fae7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb0102c70 0x7f7fb0113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50668/0 (socket says 172.21.15.79:50668) 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fae7fc700 1 -- 172.21.15.79:0/10345252 learned_addr learned my addr 172.21.15.79:0/10345252 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7faeffd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb00fb1e0 0x7f7fb0113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fae7fc700 1 -- 172.21.15.79:0/10345252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb00fb1e0 msgr2=0x7f7fb0113610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.442+0000 7f7fae7fc700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb00fb1e0 0x7f7fb0113610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.443+0000 7f7fae7fc700 1 -- 172.21.15.79:0/10345252 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa4005040 con 0x7f7fb0102c70 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.443+0000 7f7fae7fc700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb0102c70 0x7f7fb0113b90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b3b0 tx=0x7f7fa4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.443+0000 7f7f977fe700 1 -- 172.21.15.79:0/10345252 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fa400e040 con 0x7f7fb0102c70 2024-01-19T01:19:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.443+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb010d700 con 0x7f7fb0102c70 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.443+0000 7f7f977fe700 1 -- 172.21.15.79:0/10345252 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fa401c430 con 0x7f7fb0102c70 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.443+0000 7f7f977fe700 1 -- 172.21.15.79:0/10345252 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fa4013640 con 0x7f7fb0102c70 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.443+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb010dc40 con 0x7f7fb0102c70 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.444+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb0105050 con 0x7f7fb0102c70 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.444+0000 7f7f977fe700 1 -- 172.21.15.79:0/10345252 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7fa401c5a0 con 0x7f7fb0102c70 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.445+0000 7f7f977fe700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7f98068f60 0x7f7f9806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.445+0000 7f7faeffd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7f98068f60 0x7f7f9806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.445+0000 7f7f977fe700 1 -- 172.21.15.79:0/10345252 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7fa4087590 con 0x7f7fb0102c70 2024-01-19T01:19:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.445+0000 7f7faeffd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7f98068f60 0x7f7f9806b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00f9930 tx=0x7f7fa000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.449+0000 7f7f977fe700 1 -- 172.21.15.79:0/10345252 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7fa40578e0 con 0x7f7fb0102c70 2024-01-19T01:19:49.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.635+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fb0109b70 con 0x7f7fb0102c70 2024-01-19T01:19:49.636 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.636+0000 7f7f977fe700 1 -- 172.21.15.79:0/10345252 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fa4057470 con 0x7f7fb0102c70 2024-01-19T01:19:49.637 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.637+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7f98068f60 msgr2=0x7f7f9806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.637+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7f98068f60 0x7f7f9806b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00f9930 tx=0x7f7fa000b040 comp rx=0 tx=0).stop 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.637+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb0102c70 msgr2=0x7f7fb0113b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.637+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb0102c70 0x7f7fb0113b90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b3b0 tx=0x7f7fa4009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.638+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 shutdown_connections 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.638+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7f98068f60 0x7f7f9806b410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.638+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb00fb1e0 0x7f7fb0113610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.638+0000 7f7fb54cd700 1 --2- 172.21.15.79:0/10345252 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7fb0102c70 0x7f7fb0113b90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.638+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 >> 172.21.15.79:0/10345252 conn(0x7f7fb00711a0 msgr2=0x7f7fb00fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.638+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 shutdown_connections 2024-01-19T01:19:49.639 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:49.638+0000 7f7fb54cd700 1 -- 172.21.15.79:0/10345252 wait complete. 2024-01-19T01:19:49.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:49.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:19:50.366 INFO:tasks.cephadm:Setting up client nodes... 2024-01-19T01:19:50.366 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-19T01:19:50.627 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:19:50.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:50 smithi079 ceph-mon[28374]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:50.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:50 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/10345252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T01:19:50.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:50 smithi152 ceph-mon[35694]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:50.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:50 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/10345252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 -- 172.21.15.79:0/3390080186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 msgr2=0x7f1d7c0fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 --2- 172.21.15.79:0/3390080186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c0fd150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1d70004940 tx=0x7f1d70009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 -- 172.21.15.79:0/3390080186 shutdown_connections 2024-01-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 --2- 172.21.15.79:0/3390080186 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c0fd150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 --2- 172.21.15.79:0/3390080186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d7c0fbaf0 0x7f1d7c0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 -- 172.21.15.79:0/3390080186 >> 172.21.15.79:0/3390080186 conn(0x7f1d7c0f7190 msgr2=0x7f1d7c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:52.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 -- 172.21.15.79:0/3390080186 shutdown_connections 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.219+0000 7f1d80d81700 1 -- 172.21.15.79:0/3390080186 wait complete. 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d80d81700 1 Processor -- start 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d80d81700 1 -- start start 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d80d81700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d7c0fbaf0 0x7f1d7c10c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d80d81700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c10c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d80d81700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10d040 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d80d81700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10d880 con 0x7f1d7c0fbaf0 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d79d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c10c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d79d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c10c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50692/0 (socket says 172.21.15.79:50692) 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d79d9b700 1 -- 172.21.15.79:0/1976309695 learned_addr learned my addr 172.21.15.79:0/1976309695 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d7a59c700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d7c0fbaf0 0x7f1d7c10c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d79d9b700 1 -- 172.21.15.79:0/1976309695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d7c0fbaf0 msgr2=0x7f1d7c10c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d79d9b700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d7c0fbaf0 0x7f1d7c10c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d79d9b700 1 -- 172.21.15.79:0/1976309695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d70005040 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d79d9b700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c10c9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009360 tx=0x7f1d70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:52.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.220+0000 7f1d6affd700 1 -- 172.21.15.79:0/1976309695 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7000e040 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.221+0000 7f1d6affd700 1 -- 172.21.15.79:0/1976309695 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d7000bcd0 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.221+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c10db00 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.221+0000 7f1d6affd700 1 -- 172.21.15.79:0/1976309695 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d7001c630 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.224 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.221+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c10e040 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.222+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c04bb80 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.222+0000 7f1d6affd700 1 -- 172.21.15.79:0/1976309695 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1d70013960 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.222+0000 7f1d6affd700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1d64068f60 0x7f1d6406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:52.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.222+0000 7f1d7a59c700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1d64068f60 0x7f1d6406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:52.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.222+0000 7f1d6affd700 1 -- 172.21.15.79:0/1976309695 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1d70087f20 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.225 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.223+0000 7f1d7a59c700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1d64068f60 0x7f1d6406b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c006fd0 tx=0x7f1d6c008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:52.227 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.227+0000 7f1d6affd700 1 -- 172.21.15.79:0/1976309695 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1d700b9a00 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.487 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.486+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1d7c10e3f0 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.490+0000 7f1d6affd700 1 -- 172.21.15.79:0/1976309695 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1d700581c0 con 0x7f1d7c0fcd00 2024-01-19T01:19:52.490 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-01-19T01:19:52.491 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQA4zqllSL4SHRAAV+h1rLkz203geQqXHEakIA== 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.491+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1d64068f60 msgr2=0x7f1d6406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.491+0000 7f1d80d81700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1d64068f60 0x7f1d6406b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c006fd0 tx=0x7f1d6c008040 comp rx=0 tx=0).stop 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.491+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 msgr2=0x7f1d7c10c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.491+0000 7f1d80d81700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c10c9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009360 tx=0x7f1d70009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.491+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 shutdown_connections 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.491+0000 7f1d80d81700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1d64068f60 0x7f1d6406b410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.491+0000 7f1d80d81700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d7c0fbaf0 0x7f1d7c10c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.492+0000 7f1d80d81700 1 --2- 172.21.15.79:0/1976309695 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1d7c0fcd00 0x7f1d7c10c9b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.492+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 >> 172.21.15.79:0/1976309695 conn(0x7f1d7c0f7190 msgr2=0x7f1d7c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.492+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 shutdown_connections 2024-01-19T01:19:52.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:52.492+0000 7f1d80d81700 1 -- 172.21.15.79:0/1976309695 wait complete. 2024-01-19T01:19:52.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:52 smithi079 ceph-mon[28374]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:52.755 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:52 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1976309695' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T01:19:52.756 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:52 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1976309695' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T01:19:52.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:52 smithi152 ceph-mon[35694]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:52.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:52 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1976309695' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T01:19:52.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:52 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1976309695' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T01:19:53.222 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:19:53.223 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-01-19T01:19:53.223 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-19T01:19:53.265 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-19T01:19:53.533 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi152/config 2024-01-19T01:19:54.807 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:54 smithi152 ceph-mon[35694]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:54.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:54 smithi079 ceph-mon[28374]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:55.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- 172.21.15.152:0/3463614559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 msgr2=0x7f1fac0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:55.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3463614559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac0fbf00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8004940 tx=0x7f1fa8009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:55.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- 172.21.15.152:0/3463614559 shutdown_connections 2024-01-19T01:19:55.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3463614559 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fac0fcd00 0x7f1fac0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:55.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3463614559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac0fbf00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- 172.21.15.152:0/3463614559 >> 172.21.15.152:0/3463614559 conn(0x7f1fac0f71d0 msgr2=0x7f1fac0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- 172.21.15.152:0/3463614559 shutdown_connections 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- 172.21.15.152:0/3463614559 wait complete. 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 Processor -- start 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- start start 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac112680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fac0fcd00 0x7f1fac114bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac066890 con 0x7f1fac0fcd00 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb4695700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac0669d0 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.014+0000 7f1fb2431700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac112680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb2431700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac112680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:46404/0 (socket says 172.21.15.152:46404) 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb2431700 1 -- 172.21.15.152:0/3271402870 learned_addr learned my addr 172.21.15.152:0/3271402870 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb1c30700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fac0fcd00 0x7f1fac114bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb2431700 1 -- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fac0fcd00 msgr2=0x7f1fac114bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:55.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb2431700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fac0fcd00 0x7f1fac114bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb2431700 1 -- 172.21.15.152:0/3271402870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa8005040 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb1c30700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fac0fcd00 0x7f1fac114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb2431700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac112680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009360 tx=0x7f1fa8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fa2ffd700 1 -- 172.21.15.152:0/3271402870 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fa800e040 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac115110 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fa2ffd700 1 -- 172.21.15.152:0/3271402870 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fa8004290 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fa2ffd700 1 -- 172.21.15.152:0/3271402870 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fa801c630 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.015+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fac1154b0 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.016+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fac1064d0 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.017+0000 7f1fa2ffd700 1 -- 172.21.15.152:0/3271402870 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1fa8013960 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.017+0000 7f1fa2ffd700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1f98068f70 0x7f1f9806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:55.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.017+0000 7f1fa2ffd700 1 -- 172.21.15.152:0/3271402870 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1fa8087df0 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.017+0000 7f1fb1c30700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1f98068f70 0x7f1f9806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:55.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.018+0000 7f1fb1c30700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1f98068f70 0x7f1f9806b420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c009fd0 tx=0x7f1f9c009ee0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:55.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.021+0000 7f1fa2ffd700 1 -- 172.21.15.152:0/3271402870 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1fa8054600 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.278 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.277+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1fac104490 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.283 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.282+0000 7f1fa2ffd700 1 -- 172.21.15.152:0/3271402870 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1fa8057c20 con 0x7f1fac0fbaf0 2024-01-19T01:19:55.283 INFO:teuthology.orchestra.run.smithi152.stdout:[client.1] 2024-01-19T01:19:55.283 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQA7zqll6nilEBAAT+WsVnDhfCTghjR2jKMTUw== 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1f98068f70 msgr2=0x7f1f9806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1f98068f70 0x7f1f9806b420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c009fd0 tx=0x7f1f9c009ee0 comp rx=0 tx=0).stop 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 msgr2=0x7f1fac112680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac112680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009360 tx=0x7f1fa8009e30 comp rx=0 tx=0).stop 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 shutdown_connections 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1f98068f70 0x7f1f9806b420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fac0fbaf0 0x7f1fac112680 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 --2- 172.21.15.152:0/3271402870 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1fac0fcd00 0x7f1fac114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 >> 172.21.15.152:0/3271402870 conn(0x7f1fac0f71d0 msgr2=0x7f1fac0f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 shutdown_connections 2024-01-19T01:19:55.285 INFO:teuthology.orchestra.run.smithi152.stderr:2024-01-19T01:19:55.284+0000 7f1fb4695700 1 -- 172.21.15.152:0/3271402870 wait complete. 2024-01-19T01:19:55.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:55 smithi079 ceph-mon[28374]: from='client.? 172.21.15.152:0/3271402870' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T01:19:55.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:55 smithi079 ceph-mon[28374]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T01:19:55.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:55 smithi079 ceph-mon[28374]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T01:19:55.852 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:55 smithi152 ceph-mon[35694]: from='client.? 172.21.15.152:0/3271402870' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T01:19:55.852 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:55 smithi152 ceph-mon[35694]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-19T01:19:55.852 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:55 smithi152 ceph-mon[35694]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-19T01:19:55.877 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:19:55.877 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-01-19T01:19:55.877 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-19T01:19:55.922 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-19T01:19:55.922 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-01-19T01:19:55.922 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph mgr dump --format=json 2024-01-19T01:19:56.176 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:19:56.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:56 smithi152 ceph-mon[35694]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:56.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:56 smithi079 ceph-mon[28374]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 -- 172.21.15.79:0/3658404064 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 msgr2=0x7fbbc00fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/3658404064 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc00fbfb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009ce0 tx=0x7fbbb4004d10 comp rx=0 tx=0).stop 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 -- 172.21.15.79:0/3658404064 shutdown_connections 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/3658404064 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc00fbfb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/3658404064 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc0101b00 0x7fbbc0101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 -- 172.21.15.79:0/3658404064 >> 172.21.15.79:0/3658404064 conn(0x7fbbc00f76c0 msgr2=0x7fbbc00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 -- 172.21.15.79:0/3658404064 shutdown_connections 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.796+0000 7fbbc48d0700 1 -- 172.21.15.79:0/3658404064 wait complete. 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbc48d0700 1 Processor -- start 2024-01-19T01:19:57.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbc48d0700 1 -- start start 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbc48d0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc01126d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbc48d0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc0101b00 0x7fbbc0112c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbc48d0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc01132f0 con 0x7fbbc00fbb40 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbc48d0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc010c5b0 con 0x7fbbc0101b00 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbbe59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc01126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbbe59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc01126d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:50698/0 (socket says 172.21.15.79:50698) 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbbe59c700 1 -- 172.21.15.79:0/497191854 learned_addr learned my addr 172.21.15.79:0/497191854 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbbe59c700 1 -- 172.21.15.79:0/497191854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc0101b00 msgr2=0x7fbbc0112c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.797+0000 7fbbbe59c700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc0101b00 0x7fbbc0112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.798+0000 7fbbbe59c700 1 -- 172.21.15.79:0/497191854 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4009940 con 0x7fbbc00fbb40 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.798+0000 7fbbbe59c700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc01126d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800b790 tx=0x7fbbb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.798+0000 7fbbaeffd700 1 -- 172.21.15.79:0/497191854 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb800f770 con 0x7fbbc00fbb40 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.798+0000 7fbbaeffd700 1 -- 172.21.15.79:0/497191854 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbb800fd80 con 0x7fbbc00fbb40 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.798+0000 7fbbaeffd700 1 -- 172.21.15.79:0/497191854 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbbb800d310 con 0x7fbbc00fbb40 2024-01-19T01:19:57.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.798+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc010c890 con 0x7fbbc00fbb40 2024-01-19T01:19:57.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.798+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc010cdd0 con 0x7fbbc00fbb40 2024-01-19T01:19:57.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.799+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc0103fb0 con 0x7fbbc00fbb40 2024-01-19T01:19:57.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.799+0000 7fbbaeffd700 1 -- 172.21.15.79:0/497191854 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbbb8010420 con 0x7fbbc00fbb40 2024-01-19T01:19:57.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.800+0000 7fbbaeffd700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fbba8068f70 0x7fbba806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:19:57.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.800+0000 7fbbaeffd700 1 -- 172.21.15.79:0/497191854 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbbb8086a40 con 0x7fbbc00fbb40 2024-01-19T01:19:57.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.800+0000 7fbbbdd9b700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fbba8068f70 0x7fbba806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:19:57.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.800+0000 7fbbbdd9b700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fbba8068f70 0x7fbba806b420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4005010 tx=0x7fbbb4004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:19:57.806 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:57.804+0000 7fbbaeffd700 1 -- 172.21.15.79:0/497191854 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbbb8056ce0 con 0x7fbbc00fbb40 2024-01-19T01:19:58.042 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.040+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fbbc010dcf0 con 0x7fbbc00fbb40 2024-01-19T01:19:58.051 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.043+0000 7fbbaeffd700 1 -- 172.21.15.79:0/497191854 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+166774 (secure 0 0 0) 0x7fbbb8056870 con 0x7fbbc00fbb40 2024-01-19T01:19:58.052 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:19:58.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fbba8068f70 msgr2=0x7fbba806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:58.067 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fbba8068f70 0x7fbba806b420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4005010 tx=0x7fbbb4004d10 comp rx=0 tx=0).stop 2024-01-19T01:19:58.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 msgr2=0x7fbbc01126d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:19:58.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc01126d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800b790 tx=0x7fbbb8009250 comp rx=0 tx=0).stop 2024-01-19T01:19:58.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 shutdown_connections 2024-01-19T01:19:58.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fbba8068f70 0x7fbba806b420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:58.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fbbc00fbb40 0x7fbbc01126d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:58.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 --2- 172.21.15.79:0/497191854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbbc0101b00 0x7fbbc0112c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:19:58.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 >> 172.21.15.79:0/497191854 conn(0x7fbbc00f76c0 msgr2=0x7fbbc00f9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:19:58.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 shutdown_connections 2024-01-19T01:19:58.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:19:58.046+0000 7fbbc48d0700 1 -- 172.21.15.79:0/497191854 wait complete. 2024-01-19T01:19:58.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:58 smithi079 ceph-mon[28374]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:58.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:19:58 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/497191854' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-19T01:19:58.796 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":19,"active_gid":14210,"active_name":"smithi079.miqczk","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":655853372},{"type":"v1","addr":"172.21.15.79:6801","nonce":655853372}]},"active_addr":"172.21.15.79:6801/655853372","active_change":"2024-01-19T01:16:19.163021+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi152.flyowb","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.79:8443/","prometheus":"http://172.21.15.79:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":1668076868}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":1531202405}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":2535354747}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":2186703129}]}]} 2024-01-19T01:19:58.800 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-01-19T01:19:58.800 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-01-19T01:19:58.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-19T01:19:58.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:58 smithi152 ceph-mon[35694]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:19:58.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:19:58 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/497191854' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-19T01:19:59.054 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:00.867 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:00 smithi079 ceph-mon[28374]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:00.867 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:00 smithi079 ceph-mon[28374]: overall HEALTH_OK 2024-01-19T01:20:00.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 -- 172.21.15.79:0/236566258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4180fb1e0 msgr2=0x7fa4180fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:00.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 --2- 172.21.15.79:0/236566258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4180fb1e0 0x7fa4180fb630 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa40c004940 tx=0x7fa40c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:00.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 -- 172.21.15.79:0/236566258 shutdown_connections 2024-01-19T01:20:00.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 --2- 172.21.15.79:0/236566258 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4180fb1e0 0x7fa4180fb630 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:00.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 --2- 172.21.15.79:0/236566258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa418102c70 0x7fa418103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:00.875 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 -- 172.21.15.79:0/236566258 >> 172.21.15.79:0/236566258 conn(0x7fa4180711a0 msgr2=0x7fa4180715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 -- 172.21.15.79:0/236566258 shutdown_connections 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.872+0000 7fa41d140700 1 -- 172.21.15.79:0/236566258 wait complete. 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41d140700 1 Processor -- start 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41d140700 1 -- start start 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41d140700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4180fb1e0 0x7fa4181135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41d140700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa418102c70 0x7fa418113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41d140700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa418114200 con 0x7fa418102c70 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41d140700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41810d4a0 con 0x7fa4180fb1e0 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa418102c70 0x7fa418113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa418102c70 0x7fa418113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53104/0 (socket says 172.21.15.79:53104) 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.873+0000 7fa41659c700 1 -- 172.21.15.79:0/2234478509 learned_addr learned my addr 172.21.15.79:0/2234478509 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa416d9d700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4180fb1e0 0x7fa4181135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa41659c700 1 -- 172.21.15.79:0/2234478509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4180fb1e0 msgr2=0x7fa4181135e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:00.876 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa41659c700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4180fb1e0 0x7fa4181135e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa41659c700 1 -- 172.21.15.79:0/2234478509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40c005040 con 0x7fa418102c70 2024-01-19T01:20:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa41659c700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa418102c70 0x7fa418113b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00b3b0 tx=0x7fa40c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa3ff7fe700 1 -- 172.21.15.79:0/2234478509 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40c00e040 con 0x7fa418102c70 2024-01-19T01:20:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa3ff7fe700 1 -- 172.21.15.79:0/2234478509 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa40c00bc50 con 0x7fa418102c70 2024-01-19T01:20:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa3ff7fe700 1 -- 172.21.15.79:0/2234478509 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40c013420 con 0x7fa418102c70 2024-01-19T01:20:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41810d720 con 0x7fa418102c70 2024-01-19T01:20:00.877 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.874+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41810dc60 con 0x7fa418102c70 2024-01-19T01:20:00.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.875+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa418072cc0 con 0x7fa418102c70 2024-01-19T01:20:00.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.875+0000 7fa3ff7fe700 1 -- 172.21.15.79:0/2234478509 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa40c01c430 con 0x7fa418102c70 2024-01-19T01:20:00.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.876+0000 7fa3ff7fe700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa400068f60 0x7fa40006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:00.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.876+0000 7fa3ff7fe700 1 -- 172.21.15.79:0/2234478509 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa40c087740 con 0x7fa418102c70 2024-01-19T01:20:00.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.876+0000 7fa416d9d700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa400068f60 0x7fa40006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:00.878 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.876+0000 7fa416d9d700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa400068f60 0x7fa40006b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:00.881 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:00.880+0000 7fa3ff7fe700 1 -- 172.21.15.79:0/2234478509 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa40c057b30 con 0x7fa418102c70 2024-01-19T01:20:00.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:00 smithi152 ceph-mon[35694]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:00.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:00 smithi152 ceph-mon[35694]: overall HEALTH_OK 2024-01-19T01:20:01.065 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.064+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa418109bf0 con 0x7fa418102c70 2024-01-19T01:20:01.066 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.065+0000 7fa3ff7fe700 1 -- 172.21.15.79:0/2234478509 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11091 (secure 0 0 0) 0x7fa40c0576c0 con 0x7fa418102c70 2024-01-19T01:20:01.066 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:20:01.066 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","created":"2024-01-19T01:14:07.716096+0000","modified":"2024-01-19T01:19:44.537500+0000","last_up_change":"2024-01-19T01:19:43.531498+0000","last_in_change":"2024-01-19T01:19:25.050496+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T01:18:29.500877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4500f84c-0d7c-472b-8d18-35de388ec5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6803","nonce":2692943028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6805","nonce":2692943028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6809","nonce":2692943028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6807","nonce":2692943028}]},"public_addr":"172.21.15.79:6803/2692943028","cluster_addr":"172.21.15.79:6805/2692943028","heartbeat_back_addr":"172.21.15.79:6809/2692943028","heartbeat_front_addr":"172.21.15.79:6807/2692943028","state":["exists","up"]},{"osd":1,"uuid":"0286a58d-91d9-48f3-9a73-cad13c064623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6811","nonce":3254175144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6813","nonce":3254175144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6817","nonce":3254175144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6815","nonce":3254175144}]},"public_addr":"172.21.15.79:6811/3254175144","cluster_addr":"172.21.15.79:6813/3254175144","heartbeat_back_addr":"172.21.15.79:6817/3254175144","heartbeat_front_addr":"172.21.15.79:6815/3254175144","state":["exists","up"]},{"osd":2,"uuid":"1d32cd7f-64af-4edf-b9cf-3db2519b9bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6819","nonce":3125547025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6821","nonce":3125547025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6825","nonce":3125547025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6823","nonce":3125547025}]},"public_addr":"172.21.15.79:6819/3125547025","cluster_addr":"172.21.15.79:6821/3125547025","heartbeat_back_addr":"172.21.15.79:6825/3125547025","heartbeat_front_addr":"172.21.15.79:6823/3125547025","state":["exists","up"]},{"osd":3,"uuid":"4ac48076-9143-445e-bac1-0ae801b0d3fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6801","nonce":2401180651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6803","nonce":2401180651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6807","nonce":2401180651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6805","nonce":2401180651}]},"public_addr":"172.21.15.152:6801/2401180651","cluster_addr":"172.21.15.152:6803/2401180651","heartbeat_back_addr":"172.21.15.152:6807/2401180651","heartbeat_front_addr":"172.21.15.152:6805/2401180651","state":["exists","up"]},{"osd":4,"uuid":"7c505c3b-0853-4af4-bb1b-4e32ba16bede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6809","nonce":1289083445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6811","nonce":1289083445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6815","nonce":1289083445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6813","nonce":1289083445}]},"public_addr":"172.21.15.152:6809/1289083445","cluster_addr":"172.21.15.152:6811/1289083445","heartbeat_back_addr":"172.21.15.152:6815/1289083445","heartbeat_front_addr":"172.21.15.152:6813/1289083445","state":["exists","up"]},{"osd":5,"uuid":"c343c013-2f74-4c20-83bd-14e41c304d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6817","nonce":3123617721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6819","nonce":3123617721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6823","nonce":3123617721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6821","nonce":3123617721}]},"public_addr":"172.21.15.152:6817/3123617721","cluster_addr":"172.21.15.152:6819/3123617721","heartbeat_back_addr":"172.21.15.152:6823/3123617721","heartbeat_front_addr":"172.21.15.152:6821/3123617721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:17:31.789908+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:00.346643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:27.657969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:51.195401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:17.092535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:42.162295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1393070546":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6800/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/780174951":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/207141703":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/1174599613":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6801/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6800/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6801/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6801/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6800/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/2671943691":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/4252083415":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/907783696":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/2783373766":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/547589622":"2024-01-20T01:14:38.242827+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T01:20:01.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa400068f60 msgr2=0x7fa40006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:01.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa400068f60 0x7fa40006b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408008040 comp rx=0 tx=0).stop 2024-01-19T01:20:01.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa418102c70 msgr2=0x7fa418113b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:01.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa418102c70 0x7fa418113b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00b3b0 tx=0x7fa40c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:01.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 shutdown_connections 2024-01-19T01:20:01.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa400068f60 0x7fa40006b410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:01.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4180fb1e0 0x7fa4181135e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:01.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 --2- 172.21.15.79:0/2234478509 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa418102c70 0x7fa418113b20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:01.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 >> 172.21.15.79:0/2234478509 conn(0x7fa4180711a0 msgr2=0x7fa4180fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:01.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 shutdown_connections 2024-01-19T01:20:01.069 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:01.067+0000 7fa41d140700 1 -- 172.21.15.79:0/2234478509 wait complete. 2024-01-19T01:20:01.697 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-01-19T01:20:01.697 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-19T01:20:01.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:01 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2234478509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:20:01.996 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:01.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:01 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2234478509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:20:02.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:02 smithi079 ceph-mon[28374]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:02.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:02 smithi152 ceph-mon[35694]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:03.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 -- 172.21.15.79:0/3781853281 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00065390 msgr2=0x7f3d00065800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:03.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3781853281 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00065390 0x7f3d00065800 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0004940 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:03.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 -- 172.21.15.79:0/3781853281 shutdown_connections 2024-01-19T01:20:03.757 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3781853281 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00065390 0x7f3d00065800 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3781853281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d00102c20 0x7f3d00102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 -- 172.21.15.79:0/3781853281 >> 172.21.15.79:0/3781853281 conn(0x7f3d000719b0 msgr2=0x7f3d00071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 -- 172.21.15.79:0/3781853281 shutdown_connections 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 -- 172.21.15.79:0/3781853281 wait complete. 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 Processor -- start 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.755+0000 7f3d04acd700 1 -- start start 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3d04acd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d00065390 0x7f3d00113730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3d04acd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00102c20 0x7f3d00113c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3d04acd700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001142c0 con 0x7f3d00102c20 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3d04acd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0010d5f0 con 0x7f3d00065390 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00102c20 0x7f3d00113c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00102c20 0x7f3d00113c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53112/0 (socket says 172.21.15.79:53112) 2024-01-19T01:20:03.758 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfdd9b700 1 -- 172.21.15.79:0/3425475396 learned_addr learned my addr 172.21.15.79:0/3425475396 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfe59c700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d00065390 0x7f3d00113730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfdd9b700 1 -- 172.21.15.79:0/3425475396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d00065390 msgr2=0x7f3d00113730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfdd9b700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d00065390 0x7f3d00113730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfdd9b700 1 -- 172.21.15.79:0/3425475396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0005040 con 0x7f3d00102c20 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3cfdd9b700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00102c20 0x7f3d00113c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b3b0 tx=0x7f3cf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3ceeffd700 1 -- 172.21.15.79:0/3425475396 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf000e040 con 0x7f3d00102c20 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3ceeffd700 1 -- 172.21.15.79:0/3425475396 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cf001c430 con 0x7f3d00102c20 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0010d870 con 0x7f3d00102c20 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3ceeffd700 1 -- 172.21.15.79:0/3425475396 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cf0013640 con 0x7f3d00102c20 2024-01-19T01:20:03.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.756+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0010ddb0 con 0x7f3d00102c20 2024-01-19T01:20:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.758+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d000fff40 con 0x7f3d00102c20 2024-01-19T01:20:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.758+0000 7f3ceeffd700 1 -- 172.21.15.79:0/3425475396 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f3cf001c5a0 con 0x7f3d00102c20 2024-01-19T01:20:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.758+0000 7f3ceeffd700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3ce8068f60 0x7f3ce806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.758+0000 7f3ceeffd700 1 -- 172.21.15.79:0/3425475396 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3cf0087580 con 0x7f3d00102c20 2024-01-19T01:20:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.758+0000 7f3cfe59c700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3ce8068f60 0x7f3ce806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:03.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.759+0000 7f3cfe59c700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3ce8068f60 0x7f3ce806b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4007fd0 tx=0x7f3cf4009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:03.763 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.763+0000 7f3ceeffd700 1 -- 172.21.15.79:0/3425475396 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3cf00578d0 con 0x7f3d00102c20 2024-01-19T01:20:03.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.948+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d0004bb80 con 0x7f3d00102c20 2024-01-19T01:20:03.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.948+0000 7f3ceeffd700 1 -- 172.21.15.79:0/3425475396 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11091 (secure 0 0 0) 0x7f3cf0057460 con 0x7f3d00102c20 2024-01-19T01:20:03.950 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:20:03.950 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":40,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","created":"2024-01-19T01:14:07.716096+0000","modified":"2024-01-19T01:19:44.537500+0000","last_up_change":"2024-01-19T01:19:43.531498+0000","last_in_change":"2024-01-19T01:19:25.050496+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T01:18:29.500877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4500f84c-0d7c-472b-8d18-35de388ec5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6803","nonce":2692943028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6805","nonce":2692943028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6809","nonce":2692943028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6807","nonce":2692943028}]},"public_addr":"172.21.15.79:6803/2692943028","cluster_addr":"172.21.15.79:6805/2692943028","heartbeat_back_addr":"172.21.15.79:6809/2692943028","heartbeat_front_addr":"172.21.15.79:6807/2692943028","state":["exists","up"]},{"osd":1,"uuid":"0286a58d-91d9-48f3-9a73-cad13c064623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6811","nonce":3254175144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6813","nonce":3254175144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6817","nonce":3254175144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6815","nonce":3254175144}]},"public_addr":"172.21.15.79:6811/3254175144","cluster_addr":"172.21.15.79:6813/3254175144","heartbeat_back_addr":"172.21.15.79:6817/3254175144","heartbeat_front_addr":"172.21.15.79:6815/3254175144","state":["exists","up"]},{"osd":2,"uuid":"1d32cd7f-64af-4edf-b9cf-3db2519b9bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6819","nonce":3125547025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6821","nonce":3125547025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6825","nonce":3125547025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6823","nonce":3125547025}]},"public_addr":"172.21.15.79:6819/3125547025","cluster_addr":"172.21.15.79:6821/3125547025","heartbeat_back_addr":"172.21.15.79:6825/3125547025","heartbeat_front_addr":"172.21.15.79:6823/3125547025","state":["exists","up"]},{"osd":3,"uuid":"4ac48076-9143-445e-bac1-0ae801b0d3fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6801","nonce":2401180651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6803","nonce":2401180651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6807","nonce":2401180651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6805","nonce":2401180651}]},"public_addr":"172.21.15.152:6801/2401180651","cluster_addr":"172.21.15.152:6803/2401180651","heartbeat_back_addr":"172.21.15.152:6807/2401180651","heartbeat_front_addr":"172.21.15.152:6805/2401180651","state":["exists","up"]},{"osd":4,"uuid":"7c505c3b-0853-4af4-bb1b-4e32ba16bede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6809","nonce":1289083445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6811","nonce":1289083445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6815","nonce":1289083445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6813","nonce":1289083445}]},"public_addr":"172.21.15.152:6809/1289083445","cluster_addr":"172.21.15.152:6811/1289083445","heartbeat_back_addr":"172.21.15.152:6815/1289083445","heartbeat_front_addr":"172.21.15.152:6813/1289083445","state":["exists","up"]},{"osd":5,"uuid":"c343c013-2f74-4c20-83bd-14e41c304d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6817","nonce":3123617721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6819","nonce":3123617721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6823","nonce":3123617721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6821","nonce":3123617721}]},"public_addr":"172.21.15.152:6817/3123617721","cluster_addr":"172.21.15.152:6819/3123617721","heartbeat_back_addr":"172.21.15.152:6823/3123617721","heartbeat_front_addr":"172.21.15.152:6821/3123617721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:17:31.789908+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:00.346643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:27.657969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:51.195401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:17.092535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:42.162295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/1393070546":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6800/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/780174951":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/207141703":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/1174599613":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6801/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6800/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6801/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6801/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6800/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/2671943691":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/4252083415":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/907783696":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/2783373766":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/547589622":"2024-01-20T01:14:38.242827+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T01:20:03.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3ce8068f60 msgr2=0x7f3ce806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:03.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3ce8068f60 0x7f3ce806b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4007fd0 tx=0x7f3cf4009040 comp rx=0 tx=0).stop 2024-01-19T01:20:03.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00102c20 msgr2=0x7f3d00113c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:03.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00102c20 0x7f3d00113c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000b3b0 tx=0x7f3cf0009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:03.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 shutdown_connections 2024-01-19T01:20:03.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3ce8068f60 0x7f3ce806b410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:03.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d00065390 0x7f3d00113730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:03.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 --2- 172.21.15.79:0/3425475396 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3d00102c20 0x7f3d00113c70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:03.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.950+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 >> 172.21.15.79:0/3425475396 conn(0x7f3d000719b0 msgr2=0x7f3d000fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:03.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.951+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 shutdown_connections 2024-01-19T01:20:03.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:03.951+0000 7f3d04acd700 1 -- 172.21.15.79:0/3425475396 wait complete. 2024-01-19T01:20:04.673 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-01-19T01:20:04.673 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-01-19T01:20:04.674 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-01-19T01:20:04.674 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-01-19T01:20:04.675 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-01-19T01:20:04.675 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-01-19T01:20:04.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:04 smithi152 ceph-mon[35694]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:04 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3425475396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:20:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:04.979 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:04.990 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:04 smithi079 ceph-mon[28374]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:04 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3425475396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:20:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:06.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:06 smithi079 ceph-mon[28374]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:06.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:06 smithi152 ceph-mon[35694]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:07.099 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:07.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 -- 172.21.15.79:0/882823652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f940fbb40 msgr2=0x7f4f940fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:07.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 --2- 172.21.15.79:0/882823652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f940fbb40 0x7f4f940fbfb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4f90004940 tx=0x7f4f90009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 -- 172.21.15.79:0/882823652 shutdown_connections 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 --2- 172.21.15.79:0/882823652 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f940fbb40 0x7f4f940fbfb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 --2- 172.21.15.79:0/882823652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f94101b00 0x7f4f94101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 -- 172.21.15.79:0/882823652 >> 172.21.15.79:0/882823652 conn(0x7f4f940719b0 msgr2=0x7f4f94071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 -- 172.21.15.79:0/882823652 shutdown_connections 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.715+0000 7f4f9b681700 1 -- 172.21.15.79:0/882823652 wait complete. 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f9b681700 1 Processor -- start 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f9b681700 1 -- start start 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f9b681700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f940fbb40 0x7f4f9410e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:07.719 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f9b681700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f94101b00 0x7f4f9410ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f9b681700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9410f280 con 0x7f4f94101b00 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f9b681700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9410f3f0 con 0x7f4f940fbb40 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f98c1c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f94101b00 0x7f4f9410ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f98c1c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f94101b00 0x7f4f9410ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53132/0 (socket says 172.21.15.79:53132) 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f98c1c700 1 -- 172.21.15.79:0/4016846458 learned_addr learned my addr 172.21.15.79:0/4016846458 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f9941d700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f940fbb40 0x7f4f9410e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f98c1c700 1 -- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f940fbb40 msgr2=0x7f4f9410e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f98c1c700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f940fbb40 0x7f4f9410e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.720 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f98c1c700 1 -- 172.21.15.79:0/4016846458 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f90005040 con 0x7f4f94101b00 2024-01-19T01:20:07.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f98c1c700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f94101b00 0x7f4f9410ec80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000b3b0 tx=0x7f4f90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:07.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.716+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f9000e040 con 0x7f4f94101b00 2024-01-19T01:20:07.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.717+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f9000bc50 con 0x7f4f94101b00 2024-01-19T01:20:07.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.717+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f941176e0 con 0x7f4f94101b00 2024-01-19T01:20:07.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.717+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f90013420 con 0x7f4f94101b00 2024-01-19T01:20:07.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.717+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f94117bc0 con 0x7f4f94101b00 2024-01-19T01:20:07.721 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.717+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f9405e300 con 0x7f4f94101b00 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.718+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4f9001c430 con 0x7f4f94101b00 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.718+0000 7f4f89ffb700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4f80068f60 0x7f4f8006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.719+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4f90088300 con 0x7f4f94101b00 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.719+0000 7f4f9941d700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4f80068f60 0x7f4f8006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.719+0000 7f4f89ffb700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] conn(0x7f4f8006ec30 0x7f4f80071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.719+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 --> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f800716f0 con 0x7f4f8006ec30 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.719+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f90088700 con 0x7f4f94101b00 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.719+0000 7f4f9941d700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4f80068f60 0x7f4f8006b410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f840041d0 tx=0x7f4f84009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.719+0000 7f4f99c1e700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] conn(0x7f4f8006ec30 0x7f4f80071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.720+0000 7f4f99c1e700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] conn(0x7f4f8006ec30 0x7f4f80071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:07.723 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.720+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== osd.5 v2:172.21.15.152:6816/3123617721 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4f800716f0 con 0x7f4f8006ec30 2024-01-19T01:20:07.729 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:07.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.736+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 --> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4f9405e700 con 0x7f4f8006ec30 2024-01-19T01:20:07.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.736+0000 7f4f89ffb700 1 -- 172.21.15.79:0/4016846458 <== osd.5 v2:172.21.15.152:6816/3123617721 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4f9405e700 con 0x7f4f8006ec30 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.736+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] conn(0x7f4f8006ec30 msgr2=0x7f4f80071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.736+0000 7f4f9b681700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] conn(0x7f4f8006ec30 0x7f4f80071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4f80068f60 msgr2=0x7f4f8006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4f80068f60 0x7f4f8006b410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f840041d0 tx=0x7f4f84009040 comp rx=0 tx=0).stop 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f94101b00 msgr2=0x7f4f9410ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f94101b00 0x7f4f9410ec80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000b3b0 tx=0x7f4f90009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 shutdown_connections 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:6816/3123617721,v1:172.21.15.152:6817/3123617721] conn(0x7f4f8006ec30 0x7f4f80071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4f80068f60 0x7f4f8006b410 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f940fbb40 0x7f4f9410e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 --2- 172.21.15.79:0/4016846458 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f4f94101b00 0x7f4f9410ec80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:07.738 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 >> 172.21.15.79:0/4016846458 conn(0x7f4f940719b0 msgr2=0x7f4f940fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:07.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 shutdown_connections 2024-01-19T01:20:07.739 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:07.737+0000 7f4f9b681700 1 -- 172.21.15.79:0/4016846458 wait complete. 2024-01-19T01:20:08.873 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:08 smithi079 ceph-mon[28374]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:08.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:08 smithi152 ceph-mon[35694]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:10.021 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:10.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.024+0000 7efc59393700 1 -- 172.21.15.79:0/317064448 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fdc60 msgr2=0x7efc540fe030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.024+0000 7efc59393700 1 --2- 172.21.15.79:0/317064448 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fdc60 0x7efc540fe030 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efc44004940 tx=0x7efc44009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:10.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.024+0000 7efc59393700 1 -- 172.21.15.79:0/317064448 shutdown_connections 2024-01-19T01:20:10.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.024+0000 7efc59393700 1 --2- 172.21.15.79:0/317064448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc540fe600 0x7efc54106c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.024+0000 7efc59393700 1 --2- 172.21.15.79:0/317064448 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fdc60 0x7efc540fe030 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.024+0000 7efc59393700 1 -- 172.21.15.79:0/317064448 >> 172.21.15.79:0/317064448 conn(0x7efc540f7690 msgr2=0x7efc540f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:10.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.024+0000 7efc59393700 1 -- 172.21.15.79:0/317064448 shutdown_connections 2024-01-19T01:20:10.026 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.025+0000 7efc59393700 1 -- 172.21.15.79:0/317064448 wait complete. 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.025+0000 7efc59393700 1 Processor -- start 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.025+0000 7efc59393700 1 -- start start 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.025+0000 7efc59393700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc540fdc60 0x7efc540fba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.025+0000 7efc59393700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fe600 0x7efc540fbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.025+0000 7efc59393700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540fc5d0 con 0x7efc540fe600 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.025+0000 7efc59393700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540fc740 con 0x7efc540fdc60 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc527fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fe600 0x7efc540fbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc527fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fe600 0x7efc540fbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54072/0 (socket says 172.21.15.79:54072) 2024-01-19T01:20:10.028 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc527fc700 1 -- 172.21.15.79:0/3463447176 learned_addr learned my addr 172.21.15.79:0/3463447176 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc52ffd700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc540fdc60 0x7efc540fba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc52ffd700 1 -- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fe600 msgr2=0x7efc540fbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc52ffd700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fe600 0x7efc540fbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc52ffd700 1 -- 172.21.15.79:0/3463447176 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc44005040 con 0x7efc540fdc60 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc52ffd700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc540fdc60 0x7efc540fba90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efc44000c00 tx=0x7efc44009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc44021070 con 0x7efc540fdc60 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc54119870 con 0x7efc540fdc60 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.026+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc4401c400 con 0x7efc540fdc60 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.027+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc4400e070 con 0x7efc540fdc60 2024-01-19T01:20:10.029 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.027+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc54119cd0 con 0x7efc540fdc60 2024-01-19T01:20:10.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.027+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc5405e300 con 0x7efc540fdc60 2024-01-19T01:20:10.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.028+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7efc44013a50 con 0x7efc540fdc60 2024-01-19T01:20:10.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.028+0000 7efc3b7fe700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7efc3c068f60 0x7efc3c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efc44087b90 con 0x7efc540fdc60 2024-01-19T01:20:10.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc527fc700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7efc3c068f60 0x7efc3c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:10.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc3b7fe700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] conn(0x7efc3c06ec30 0x7efc3c071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.031 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 --> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc3c0716f0 con 0x7efc3c06ec30 2024-01-19T01:20:10.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc4408c220 con 0x7efc540fdc60 2024-01-19T01:20:10.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc537fe700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] conn(0x7efc3c06ec30 0x7efc3c071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:10.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc527fc700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7efc3c068f60 0x7efc3c06b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efc48008fd0 tx=0x7efc4800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:10.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.029+0000 7efc537fe700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] conn(0x7efc3c06ec30 0x7efc3c071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:10.032 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.030+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== osd.2 v2:172.21.15.79:6818/3125547025 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7efc3c0716f0 con 0x7efc3c06ec30 2024-01-19T01:20:10.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.046+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 --> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efc5405e6b0 con 0x7efc3c06ec30 2024-01-19T01:20:10.047 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.046+0000 7efc3b7fe700 1 -- 172.21.15.79:0/3463447176 <== osd.2 v2:172.21.15.79:6818/3125547025 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efc5405e6b0 con 0x7efc3c06ec30 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] conn(0x7efc3c06ec30 msgr2=0x7efc3c071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] conn(0x7efc3c06ec30 0x7efc3c071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7efc3c068f60 msgr2=0x7efc3c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7efc3c068f60 0x7efc3c06b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efc48008fd0 tx=0x7efc4800b040 comp rx=0 tx=0).stop 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc540fdc60 msgr2=0x7efc540fba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc540fdc60 0x7efc540fba90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efc44000c00 tx=0x7efc44009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 shutdown_connections 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6818/3125547025,v1:172.21.15.79:6819/3125547025] conn(0x7efc3c06ec30 0x7efc3c071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7efc3c068f60 0x7efc3c06b410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc540fdc60 0x7efc540fba90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 --2- 172.21.15.79:0/3463447176 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7efc540fe600 0x7efc540fbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 >> 172.21.15.79:0/3463447176 conn(0x7efc540f7690 msgr2=0x7efc541013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:10.049 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 shutdown_connections 2024-01-19T01:20:10.050 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.047+0000 7efc59393700 1 -- 172.21.15.79:0/3463447176 wait complete. 2024-01-19T01:20:10.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 -- 172.21.15.79:0/2883466887 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c102c70 msgr2=0x7fcc3c103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 --2- 172.21.15.79:0/2883466887 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c102c70 0x7fcc3c103040 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009ce0 tx=0x7fcc34004d10 comp rx=0 tx=0).stop 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 -- 172.21.15.79:0/2883466887 shutdown_connections 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 --2- 172.21.15.79:0/2883466887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc3c0fb1e0 0x7fcc3c0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 --2- 172.21.15.79:0/2883466887 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c102c70 0x7fcc3c103040 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 -- 172.21.15.79:0/2883466887 >> 172.21.15.79:0/2883466887 conn(0x7fcc3c0711a0 msgr2=0x7fcc3c0715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 -- 172.21.15.79:0/2883466887 shutdown_connections 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.693+0000 7fcc419f4700 1 -- 172.21.15.79:0/2883466887 wait complete. 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc419f4700 1 Processor -- start 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc419f4700 1 -- start start 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc419f4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c0fb1e0 0x7fcc3c113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc419f4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc3c102c70 0x7fcc3c113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc419f4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c114320 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc419f4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c10d650 con 0x7fcc3c102c70 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc3affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c0fb1e0 0x7fcc3c113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:10.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc3affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c0fb1e0 0x7fcc3c113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54096/0 (socket says 172.21.15.79:54096) 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc3affd700 1 -- 172.21.15.79:0/1410279440 learned_addr learned my addr 172.21.15.79:0/1410279440 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc3affd700 1 -- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc3c102c70 msgr2=0x7fcc3c113cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc3affd700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc3c102c70 0x7fcc3c113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc3affd700 1 -- 172.21.15.79:0/1410279440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc34009940 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc3affd700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c0fb1e0 0x7fcc3c113790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400bee0 tx=0x7fcc34004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.694+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3400e040 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.695+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc34008f00 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.695+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c10d8d0 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.695+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc3401c710 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.695+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c10de10 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.695+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcc3c10e340 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.696+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcc34012b50 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.696+0000 7fcc337fe700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc28068f60 0x7fcc2806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.696+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcc34088670 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.696+0000 7fcc3a7fc700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc28068f60 0x7fcc2806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:10.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.696+0000 7fcc337fe700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] conn(0x7fcc2806ec30 0x7fcc28071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:10.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.696+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 --> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc280716f0 con 0x7fcc2806ec30 2024-01-19T01:20:10.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.696+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc3408ac60 con 0x7fcc3c0fb1e0 2024-01-19T01:20:10.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.697+0000 7fcc3a7fc700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc28068f60 0x7fcc2806b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c10e860 tx=0x7fcc24007d40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:10.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.697+0000 7fcc3b7fe700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] conn(0x7fcc2806ec30 0x7fcc28071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:10.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.697+0000 7fcc3b7fe700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] conn(0x7fcc2806ec30 0x7fcc28071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:10.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.698+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== osd.4 v2:172.21.15.152:6808/1289083445 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fcc280716f0 con 0x7fcc2806ec30 2024-01-19T01:20:10.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.714+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 --> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcc3c05e2c0 con 0x7fcc2806ec30 2024-01-19T01:20:10.715 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc337fe700 1 -- 172.21.15.79:0/1410279440 <== osd.4 v2:172.21.15.152:6808/1289083445 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcc3c05e2c0 con 0x7fcc2806ec30 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] conn(0x7fcc2806ec30 msgr2=0x7fcc28071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] conn(0x7fcc2806ec30 0x7fcc28071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc28068f60 msgr2=0x7fcc2806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc28068f60 0x7fcc2806b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c10e860 tx=0x7fcc24007d40 comp rx=0 tx=0).stop 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c0fb1e0 msgr2=0x7fcc3c113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c0fb1e0 0x7fcc3c113790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400bee0 tx=0x7fcc34004d10 comp rx=0 tx=0).stop 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 shutdown_connections 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:6808/1289083445,v1:172.21.15.152:6809/1289083445] conn(0x7fcc2806ec30 0x7fcc28071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc28068f60 0x7fcc2806b410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc3c0fb1e0 0x7fcc3c113790 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 --2- 172.21.15.79:0/1410279440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc3c102c70 0x7fcc3c113cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 >> 172.21.15.79:0/1410279440 conn(0x7fcc3c0711a0 msgr2=0x7fcc3c0fa180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:10.717 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.715+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 shutdown_connections 2024-01-19T01:20:10.718 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:10.716+0000 7fcc419f4700 1 -- 172.21.15.79:0/1410279440 wait complete. 2024-01-19T01:20:10.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:10 smithi152 ceph-mon[35694]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:10 smithi079 ceph-mon[28374]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:12.003 INFO:teuthology.orchestra.run.smithi079.stdout:81604378648 2024-01-19T01:20:12.004 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-01-19T01:20:12.164 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:12.187 INFO:teuthology.orchestra.run.smithi079.stdout:167503724551 2024-01-19T01:20:12.187 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-01-19T01:20:12.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:12 smithi152 ceph-mon[35694]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:12 smithi079 ceph-mon[28374]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:13.581 INFO:teuthology.orchestra.run.smithi079.stdout:146028888077 2024-01-19T01:20:13.581 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-01-19T01:20:14.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:14 smithi152 ceph-mon[35694]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:14.948 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:14 smithi079 ceph-mon[28374]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 -- 172.21.15.79:0/1804841170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d78101b00 msgr2=0x7f6d78101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/1804841170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d78101b00 0x7f6d78101ed0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6d68004940 tx=0x7f6d68009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 -- 172.21.15.79:0/1804841170 shutdown_connections 2024-01-19T01:20:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/1804841170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d780fbb40 0x7f6d780fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/1804841170 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d78101b00 0x7f6d78101ed0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.965 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 -- 172.21.15.79:0/1804841170 >> 172.21.15.79:0/1804841170 conn(0x7f6d780f76c0 msgr2=0x7f6d780f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 -- 172.21.15.79:0/1804841170 shutdown_connections 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.947+0000 7f6d7e39b700 1 -- 172.21.15.79:0/1804841170 wait complete. 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d7e39b700 1 Processor -- start 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d7e39b700 1 -- start start 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d7e39b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d780fbb40 0x7f6d78113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d7e39b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d78101b00 0x7f6d78113bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d7e39b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7810d530 con 0x7f6d780fbb40 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d7e39b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7810d6a0 con 0x7f6d78101b00 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d77fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d780fbb40 0x7f6d78113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d77fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d780fbb40 0x7f6d78113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54116/0 (socket says 172.21.15.79:54116) 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d77fff700 1 -- 172.21.15.79:0/2556012840 learned_addr learned my addr 172.21.15.79:0/2556012840 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.948+0000 7f6d777fe700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d78101b00 0x7f6d78113bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d77fff700 1 -- 172.21.15.79:0/2556012840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d78101b00 msgr2=0x7f6d78113bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d77fff700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d78101b00 0x7f6d78113bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.966 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d77fff700 1 -- 172.21.15.79:0/2556012840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d68005040 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d777fe700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d78101b00 0x7f6d78113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d77fff700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d780fbb40 0x7f6d78113670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6d68009fd0 tx=0x7f6d68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d6800e040 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d6800bc50 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7810d920 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d68013ea0 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.949+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7810de60 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.950+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6d7810e300 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.950+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6d68013420 con 0x7f6d780fbb40 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d74ff9700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6d60068f70 0x7f6d6006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:14.967 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6d68088a70 con 0x7f6d780fbb40 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d777fe700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6d60068f70 0x7f6d6006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d74ff9700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] conn(0x7f6d6006ec40 0x7f6d60071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 --> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d60071700 con 0x7f6d6006ec40 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d68088e50 con 0x7f6d780fbb40 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d7c938700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] conn(0x7f6d6006ec40 0x7f6d60071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d777fe700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6d60068f70 0x7f6d6006b420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d7810ef10 tx=0x7f6d6c008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.951+0000 7f6d7c938700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] conn(0x7f6d6006ec40 0x7f6d60071050 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:14.968 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.952+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== osd.1 v2:172.21.15.79:6810/3254175144 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6d60071700 con 0x7f6d6006ec40 2024-01-19T01:20:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 --> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6d780666b0 con 0x7f6d6006ec40 2024-01-19T01:20:14.969 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d74ff9700 1 -- 172.21.15.79:0/2556012840 <== osd.1 v2:172.21.15.79:6810/3254175144 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6d780666b0 con 0x7f6d6006ec40 2024-01-19T01:20:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] conn(0x7f6d6006ec40 msgr2=0x7f6d60071050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] conn(0x7f6d6006ec40 0x7f6d60071050 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6d60068f70 msgr2=0x7f6d6006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6d60068f70 0x7f6d6006b420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d7810ef10 tx=0x7f6d6c008040 comp rx=0 tx=0).stop 2024-01-19T01:20:14.970 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d780fbb40 msgr2=0x7f6d78113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d780fbb40 0x7f6d78113670 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6d68009fd0 tx=0x7f6d68009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 shutdown_connections 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6d60068f70 0x7f6d6006b420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6d780fbb40 0x7f6d78113670 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.79:6810/3254175144,v1:172.21.15.79:6811/3254175144] conn(0x7f6d6006ec40 0x7f6d60071050 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 --2- 172.21.15.79:0/2556012840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d78101b00 0x7f6d78113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.968+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 >> 172.21.15.79:0/2556012840 conn(0x7f6d780f76c0 msgr2=0x7f6d78065ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.969+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 shutdown_connections 2024-01-19T01:20:14.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:14.969+0000 7f6d7e39b700 1 -- 172.21.15.79:0/2556012840 wait complete. 2024-01-19T01:20:15.689 INFO:teuthology.orchestra.run.smithi079.stdout:60129542174 2024-01-19T01:20:15.690 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 -- 172.21.15.79:0/1559114234 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c0fb1e0 msgr2=0x7fa08c0fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1559114234 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c0fb1e0 0x7fa08c0fb630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa080004940 tx=0x7fa080009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 -- 172.21.15.79:0/1559114234 shutdown_connections 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1559114234 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c0fb1e0 0x7fa08c0fb630 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1559114234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa08c102c70 0x7fa08c103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 -- 172.21.15.79:0/1559114234 >> 172.21.15.79:0/1559114234 conn(0x7fa08c0711a0 msgr2=0x7fa08c0715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 -- 172.21.15.79:0/1559114234 shutdown_connections 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.918+0000 7fa090e5d700 1 -- 172.21.15.79:0/1559114234 wait complete. 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa090e5d700 1 Processor -- start 2024-01-19T01:20:15.920 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa090e5d700 1 -- start start 2024-01-19T01:20:15.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa090e5d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa08c0fb1e0 0x7fa08c113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:15.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa090e5d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c102c70 0x7fa08c113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:15.921 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa090e5d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c114200 con 0x7fa08c102c70 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa090e5d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c10d4f0 con 0x7fa08c0fb1e0 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa089d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c102c70 0x7fa08c113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa089d9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c102c70 0x7fa08c113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:54140/0 (socket says 172.21.15.79:54140) 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.919+0000 7fa089d9b700 1 -- 172.21.15.79:0/1308293825 learned_addr learned my addr 172.21.15.79:0/1308293825 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa08a59c700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa08c0fb1e0 0x7fa08c113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa089d9b700 1 -- 172.21.15.79:0/1308293825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa08c0fb1e0 msgr2=0x7fa08c113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa089d9b700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa08c0fb1e0 0x7fa08c113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa089d9b700 1 -- 172.21.15.79:0/1308293825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080005040 con 0x7fa08c102c70 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa089d9b700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c102c70 0x7fa08c113bb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa08000b3b0 tx=0x7fa080009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa08000e040 con 0x7fa08c102c70 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa08001c430 con 0x7fa08c102c70 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c10d720 con 0x7fa08c102c70 2024-01-19T01:20:15.922 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.920+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa080013640 con 0x7fa08c102c70 2024-01-19T01:20:15.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.921+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c10dc60 con 0x7fa08c102c70 2024-01-19T01:20:15.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.922+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa08c10e0f0 con 0x7fa08c102c70 2024-01-19T01:20:15.923 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.922+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa08001c5a0 con 0x7fa08c102c70 2024-01-19T01:20:15.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa07affd700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa074068f60 0x7fa07406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:15.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa080087450 con 0x7fa08c102c70 2024-01-19T01:20:15.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa08a59c700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa074068f60 0x7fa07406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:15.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa07affd700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] conn(0x7fa07406ec30 0x7fa074071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:15.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 --> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0740716f0 con 0x7fa07406ec30 2024-01-19T01:20:15.924 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa080089ac0 con 0x7fa08c102c70 2024-01-19T01:20:15.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa08ad9d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] conn(0x7fa07406ec30 0x7fa074071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:15.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.923+0000 7fa08a59c700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa074068f60 0x7fa07406b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0f9a10 tx=0x7fa07c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:15.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.924+0000 7fa08ad9d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] conn(0x7fa07406ec30 0x7fa074071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:15.925 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.925+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== osd.0 v2:172.21.15.79:6802/2692943028 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa0740716f0 con 0x7fa07406ec30 2024-01-19T01:20:15.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.941+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 --> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa08c072cc0 con 0x7fa07406ec30 2024-01-19T01:20:15.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.941+0000 7fa07affd700 1 -- 172.21.15.79:0/1308293825 <== osd.0 v2:172.21.15.79:6802/2692943028 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa08c072cc0 con 0x7fa07406ec30 2024-01-19T01:20:15.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] conn(0x7fa07406ec30 msgr2=0x7fa074071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:15.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] conn(0x7fa07406ec30 0x7fa074071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa074068f60 msgr2=0x7fa07406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa074068f60 0x7fa07406b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0f9a10 tx=0x7fa07c00b040 comp rx=0 tx=0).stop 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c102c70 msgr2=0x7fa08c113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c102c70 0x7fa08c113bb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa08000b3b0 tx=0x7fa080009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 shutdown_connections 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa074068f60 0x7fa07406b410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa08c0fb1e0 0x7fa08c113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:6802/2692943028,v1:172.21.15.79:6803/2692943028] conn(0x7fa07406ec30 0x7fa074071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 --2- 172.21.15.79:0/1308293825 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa08c102c70 0x7fa08c113bb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:15.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 >> 172.21.15.79:0/1308293825 conn(0x7fa08c0711a0 msgr2=0x7fa08c0fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:15.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 shutdown_connections 2024-01-19T01:20:15.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:15.942+0000 7fa090e5d700 1 -- 172.21.15.79:0/1308293825 wait complete. 2024-01-19T01:20:16.637 INFO:teuthology.orchestra.run.smithi079.stdout:38654705700 2024-01-19T01:20:16.638 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-01-19T01:20:16.702 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:16.703 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:16 smithi152 ceph-mon[35694]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:16.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:16 smithi079 ceph-mon[28374]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:17.181 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:17.182 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:18.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:18 smithi152 ceph-mon[35694]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:18.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:18 smithi079 ceph-mon[28374]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:19.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:19.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:20.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:20 smithi079 ceph-mon[28374]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:20.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:20 smithi152 ceph-mon[35694]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 -- 172.21.15.79:0/3573428440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9678102c00 msgr2=0x7f9678102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 --2- 172.21.15.79:0/3573428440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9678102c00 0x7f9678102fd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9674009ce0 tx=0x7f9674004d10 comp rx=0 tx=0).stop 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 -- 172.21.15.79:0/3573428440 shutdown_connections 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 --2- 172.21.15.79:0/3573428440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96781035a0 0x7f96780fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 --2- 172.21.15.79:0/3573428440 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9678102c00 0x7f9678102fd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 -- 172.21.15.79:0/3573428440 >> 172.21.15.79:0/3573428440 conn(0x7f96780f8790 msgr2=0x7f96780faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 -- 172.21.15.79:0/3573428440 shutdown_connections 2024-01-19T01:20:20.927 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.388+0000 7f967fab0700 1 -- 172.21.15.79:0/3573428440 wait complete. 2024-01-19T01:20:20.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.389+0000 7f967fab0700 1 Processor -- start 2024-01-19T01:20:20.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.389+0000 7f967fab0700 1 -- start start 2024-01-19T01:20:20.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.389+0000 7f967fab0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9678102c00 0x7f96781159b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:20.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.389+0000 7f967fab0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96781035a0 0x7f9678115ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:20.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.389+0000 7f967fab0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678116540 con 0x7f96781035a0 2024-01-19T01:20:20.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.389+0000 7f967fab0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967810f870 con 0x7f9678102c00 2024-01-19T01:20:20.928 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.389+0000 7f967d04b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96781035a0 0x7f9678115ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967d04b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96781035a0 0x7f9678115ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51380/0 (socket says 172.21.15.79:51380) 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967d04b700 1 -- 172.21.15.79:0/3655559158 learned_addr learned my addr 172.21.15.79:0/3655559158 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967d84c700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9678102c00 0x7f96781159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967d04b700 1 -- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9678102c00 msgr2=0x7f96781159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967d04b700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9678102c00 0x7f96781159b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967d04b700 1 -- 172.21.15.79:0/3655559158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674009940 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967d04b700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96781035a0 0x7f9678115ef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f966800c800 tx=0x7f9668004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96680106f0 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9668010d00 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967810fb50 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96680113a0 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.390+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9678110090 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.391+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9678110610 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.392+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9668012400 con 0x7f96781035a0 2024-01-19T01:20:20.929 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.392+0000 7f966e7fc700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9664068f60 0x7f966406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.392+0000 7f967d84c700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9664068f60 0x7f966406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.392+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f96680869d0 con 0x7f96781035a0 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.392+0000 7f966e7fc700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] conn(0x7f966406ec30 0x7f9664071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.392+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 --> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96640716f0 con 0x7f966406ec30 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.392+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f966808b220 con 0x7f96781035a0 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.393+0000 7f967e04d700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] conn(0x7f966406ec30 0x7f9664071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.393+0000 7f967d84c700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9664068f60 0x7f966406b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9674015040 tx=0x7f9674004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.393+0000 7f967e04d700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] conn(0x7f966406ec30 0x7f9664071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.394+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== osd.3 v2:172.21.15.152:6800/2401180651 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f96640716f0 con 0x7f966406ec30 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.408+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 --> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f96780faba0 con 0x7f966406ec30 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f966e7fc700 1 -- 172.21.15.79:0/3655559158 <== osd.3 v2:172.21.15.152:6800/2401180651 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f96780faba0 con 0x7f966406ec30 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] conn(0x7f966406ec30 msgr2=0x7f9664071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] conn(0x7f966406ec30 0x7f9664071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9664068f60 msgr2=0x7f966406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:20.930 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9664068f60 0x7f966406b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9674015040 tx=0x7f9674004d10 comp rx=0 tx=0).stop 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96781035a0 msgr2=0x7f9678115ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96781035a0 0x7f9678115ef0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f966800c800 tx=0x7f9668004470 comp rx=0 tx=0).stop 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 shutdown_connections 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:6800/2401180651,v1:172.21.15.152:6801/2401180651] conn(0x7f966406ec30 0x7f9664071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9664068f60 0x7f966406b410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9678102c00 0x7f96781159b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 --2- 172.21.15.79:0/3655559158 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f96781035a0 0x7f9678115ef0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.409+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 >> 172.21.15.79:0/3655559158 conn(0x7f96780f8790 msgr2=0x7f96780fa1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.410+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 shutdown_connections 2024-01-19T01:20:20.931 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:20.410+0000 7f967fab0700 1 -- 172.21.15.79:0/3655559158 wait complete. 2024-01-19T01:20:21.649 INFO:teuthology.orchestra.run.smithi079.stdout:111669149717 2024-01-19T01:20:21.650 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-01-19T01:20:22.388 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:22 smithi152 ceph-mon[35694]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:22.939 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:22 smithi079 ceph-mon[28374]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:24.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.159+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2947685747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 msgr2=0x7fa4c40fb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.159+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2947685747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fb680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4004940 tx=0x7fa4b4009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:24.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.159+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2947685747 shutdown_connections 2024-01-19T01:20:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.159+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2947685747 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fb680 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.159+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2947685747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4c4102c90 0x7fa4c4103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.159+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2947685747 >> 172.21.15.79:0/2947685747 conn(0x7fa4c40711b0 msgr2=0x7fa4c40715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.160+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2947685747 shutdown_connections 2024-01-19T01:20:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.160+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2947685747 wait complete. 2024-01-19T01:20:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.160+0000 7fa4cb37e700 1 Processor -- start 2024-01-19T01:20:24.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.160+0000 7fa4cb37e700 1 -- start start 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4cb37e700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4cb37e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4c4102c90 0x7fa4c40fc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4cb37e700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40fcfc0 con 0x7fa4c40fb210 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4cb37e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40fd130 con 0x7fa4c4102c90 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c911a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c911a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51392/0 (socket says 172.21.15.79:51392) 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c911a700 1 -- 172.21.15.79:0/2748743971 learned_addr learned my addr 172.21.15.79:0/2748743971 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c8919700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4c4102c90 0x7fa4c40fc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:24.163 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c911a700 1 -- 172.21.15.79:0/2748743971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4c4102c90 msgr2=0x7fa4c40fc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c911a700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4c4102c90 0x7fa4c40fc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c911a700 1 -- 172.21.15.79:0/2748743971 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b4005040 con 0x7fa4c40fb210 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c8919700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4c4102c90 0x7fa4c40fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.161+0000 7fa4c911a700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fc3f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000b790 tx=0x7fa4c0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.162+0000 7fa4b9ffb700 1 -- 172.21.15.79:0/2748743971 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c000f6f0 con 0x7fa4c40fb210 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.162+0000 7fa4b9ffb700 1 -- 172.21.15.79:0/2748743971 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4c000fd00 con 0x7fa4c40fb210 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.162+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c4119980 con 0x7fa4c40fb210 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.162+0000 7fa4b9ffb700 1 -- 172.21.15.79:0/2748743971 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c00103a0 con 0x7fa4c40fb210 2024-01-19T01:20:24.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.162+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c4119e70 con 0x7fa4c40fb210 2024-01-19T01:20:24.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.163+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c40fff40 con 0x7fa4c40fb210 2024-01-19T01:20:24.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.163+0000 7fa4b9ffb700 1 -- 172.21.15.79:0/2748743971 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa4c000f850 con 0x7fa4c40fb210 2024-01-19T01:20:24.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.163+0000 7fa4b9ffb700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa4b0068f60 0x7fa4b006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:24.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.164+0000 7fa4b9ffb700 1 -- 172.21.15.79:0/2748743971 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa4c0086810 con 0x7fa4c40fb210 2024-01-19T01:20:24.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.164+0000 7fa4c8919700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa4b0068f60 0x7fa4b006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:24.165 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.164+0000 7fa4c8919700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa4b0068f60 0x7fa4b006b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400b3b0 tx=0x7fa4b4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:24.169 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.168+0000 7fa4b9ffb700 1 -- 172.21.15.79:0/2748743971 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa4c0053020 con 0x7fa4c40fb210 2024-01-19T01:20:24.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.358+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa4c4075170 con 0x7fa4c40fb210 2024-01-19T01:20:24.360 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.359+0000 7fa4b9ffb700 1 -- 172.21.15.79:0/2748743971 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa4c0056640 con 0x7fa4c40fb210 2024-01-19T01:20:24.360 INFO:teuthology.orchestra.run.smithi079.stdout:146028888079 2024-01-19T01:20:24.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa4b0068f60 msgr2=0x7fa4b006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa4b0068f60 0x7fa4b006b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa4b400b3b0 tx=0x7fa4b4009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:24.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 msgr2=0x7fa4c40fc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fc3f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000b790 tx=0x7fa4c0009220 comp rx=0 tx=0).stop 2024-01-19T01:20:24.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 shutdown_connections 2024-01-19T01:20:24.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fa4b0068f60 0x7fa4b006b410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.362 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fa4c40fb210 0x7fa4c40fc3f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 --2- 172.21.15.79:0/2748743971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4c4102c90 0x7fa4c40fc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 >> 172.21.15.79:0/2748743971 conn(0x7fa4c40711b0 msgr2=0x7fa4c40fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:24.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 shutdown_connections 2024-01-19T01:20:24.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.361+0000 7fa4cb37e700 1 -- 172.21.15.79:0/2748743971 wait complete. 2024-01-19T01:20:24.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 -- 172.21.15.79:0/1140341419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba81057c0 msgr2=0x7f1ba80fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 --2- 172.21.15.79:0/1140341419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba81057c0 0x7f1ba80fee30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1b98004940 tx=0x7f1b98009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:24.396 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 -- 172.21.15.79:0/1140341419 shutdown_connections 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 --2- 172.21.15.79:0/1140341419 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba81057c0 0x7f1ba80fee30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 --2- 172.21.15.79:0/1140341419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ba8104e20 0x7f1ba81051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 -- 172.21.15.79:0/1140341419 >> 172.21.15.79:0/1140341419 conn(0x7f1ba80fa990 msgr2=0x7f1ba80fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 -- 172.21.15.79:0/1140341419 shutdown_connections 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.391+0000 7f1bb075c700 1 -- 172.21.15.79:0/1140341419 wait complete. 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bb075c700 1 Processor -- start 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bb075c700 1 -- start start 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bb075c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ba8104e20 0x7f1ba81159d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bb075c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba8115f10 0x7f1ba810f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bb075c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8116440 con 0x7f1ba8115f10 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bb075c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba81165b0 con 0x7f1ba8104e20 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1badcf7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba8115f10 0x7f1ba810f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1badcf7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba8115f10 0x7f1ba810f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51410/0 (socket says 172.21.15.79:51410) 2024-01-19T01:20:24.397 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1badcf7700 1 -- 172.21.15.79:0/3206122648 learned_addr learned my addr 172.21.15.79:0/3206122648 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bae4f8700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ba8104e20 0x7f1ba81159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1badcf7700 1 -- 172.21.15.79:0/3206122648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ba8104e20 msgr2=0x7f1ba81159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1badcf7700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ba8104e20 0x7f1ba81159d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1badcf7700 1 -- 172.21.15.79:0/3206122648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98005040 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1bae4f8700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ba8104e20 0x7f1ba81159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.392+0000 7f1badcf7700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba8115f10 0x7f1ba810f890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b98000c00 tx=0x7f1b98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.393+0000 7f1b9effd700 1 -- 172.21.15.79:0/3206122648 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b98021070 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.393+0000 7f1b9effd700 1 -- 172.21.15.79:0/3206122648 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b980093e0 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.393+0000 7f1b9effd700 1 -- 172.21.15.79:0/3206122648 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9800e070 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.393+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba810fe30 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.393+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba8110370 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.394+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba81072f0 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.394+0000 7f1b9effd700 1 -- 172.21.15.79:0/3206122648 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1b9801c690 con 0x7f1ba8115f10 2024-01-19T01:20:24.398 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.395+0000 7f1b9effd700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1b94068f60 0x7f1b9406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:24.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.395+0000 7f1b9effd700 1 -- 172.21.15.79:0/3206122648 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1b98087a30 con 0x7f1ba8115f10 2024-01-19T01:20:24.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.395+0000 7f1bae4f8700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1b94068f60 0x7f1b9406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:24.399 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.395+0000 7f1bae4f8700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1b94068f60 0x7f1b9406b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004600 tx=0x7f1ba400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:24.400 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.399+0000 7f1b9effd700 1 -- 172.21.15.79:0/3206122648 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1b98057d70 con 0x7f1ba8115f10 2024-01-19T01:20:24.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.586+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1ba8110fe0 con 0x7f1ba8115f10 2024-01-19T01:20:24.588 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.587+0000 7f1b9effd700 1 -- 172.21.15.79:0/3206122648 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1b98057d70 con 0x7f1ba8115f10 2024-01-19T01:20:24.588 INFO:teuthology.orchestra.run.smithi079.stdout:167503724554 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1b94068f60 msgr2=0x7f1b9406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1b94068f60 0x7f1b9406b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4004600 tx=0x7f1ba400b040 comp rx=0 tx=0).stop 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba8115f10 msgr2=0x7f1ba810f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba8115f10 0x7f1ba810f890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b98000c00 tx=0x7f1b98009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 shutdown_connections 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1b94068f60 0x7f1b9406b410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ba8104e20 0x7f1ba81159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 --2- 172.21.15.79:0/3206122648 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ba8115f10 0x7f1ba810f890 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:24.590 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 >> 172.21.15.79:0/3206122648 conn(0x7f1ba80fa990 msgr2=0x7f1ba80fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:24.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 shutdown_connections 2024-01-19T01:20:24.591 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:24.589+0000 7f1bb075c700 1 -- 172.21.15.79:0/3206122648 wait complete. 2024-01-19T01:20:24.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:24 smithi079 ceph-mon[28374]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:24.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:24 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2748743971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-19T01:20:24.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:24 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3206122648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-19T01:20:24.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:24 smithi152 ceph-mon[35694]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:24.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:24 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2748743971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-19T01:20:24.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:24 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3206122648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-19T01:20:25.034 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.029+0000 7fc8d5937700 1 -- 172.21.15.79:0/3670217380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d0101b00 msgr2=0x7fc8d0101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.029+0000 7fc8d5937700 1 --2- 172.21.15.79:0/3670217380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d0101b00 0x7fc8d0101ed0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc8c0004940 tx=0x7fc8c0009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.029+0000 7fc8d5937700 1 -- 172.21.15.79:0/3670217380 shutdown_connections 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.029+0000 7fc8d5937700 1 --2- 172.21.15.79:0/3670217380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8d00fbb40 0x7fc8d00fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.029+0000 7fc8d5937700 1 --2- 172.21.15.79:0/3670217380 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d0101b00 0x7fc8d0101ed0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.029+0000 7fc8d5937700 1 -- 172.21.15.79:0/3670217380 >> 172.21.15.79:0/3670217380 conn(0x7fc8d00f76c0 msgr2=0x7fc8d00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 -- 172.21.15.79:0/3670217380 shutdown_connections 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 -- 172.21.15.79:0/3670217380 wait complete. 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 Processor -- start 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 -- start start 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d00fbb40 0x7fc8d01136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8d0101b00 0x7fc8d0113bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d0114240 con 0x7fc8d00fbb40 2024-01-19T01:20:25.035 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8d5937700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d010d530 con 0x7fc8d0101b00 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8ceffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d00fbb40 0x7fc8d01136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.030+0000 7fc8ceffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d00fbb40 0x7fc8d01136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51414/0 (socket says 172.21.15.79:51414) 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8ceffd700 1 -- 172.21.15.79:0/777896302 learned_addr learned my addr 172.21.15.79:0/777896302 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8ce7fc700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8d0101b00 0x7fc8d0113bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8ceffd700 1 -- 172.21.15.79:0/777896302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8d0101b00 msgr2=0x7fc8d0113bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8ceffd700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8d0101b00 0x7fc8d0113bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8ceffd700 1 -- 172.21.15.79:0/777896302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0005040 con 0x7fc8d00fbb40 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8ce7fc700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8d0101b00 0x7fc8d0113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8ceffd700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d00fbb40 0x7fc8d01136b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000b3b0 tx=0x7fc8c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8b77fe700 1 -- 172.21.15.79:0/777896302 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8c000e040 con 0x7fc8d00fbb40 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8b77fe700 1 -- 172.21.15.79:0/777896302 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8c000bc50 con 0x7fc8d00fbb40 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8b77fe700 1 -- 172.21.15.79:0/777896302 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8c0013420 con 0x7fc8d00fbb40 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d010d7b0 con 0x7fc8d00fbb40 2024-01-19T01:20:25.036 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.031+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d010dcf0 con 0x7fc8d00fbb40 2024-01-19T01:20:25.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.032+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d0113360 con 0x7fc8d00fbb40 2024-01-19T01:20:25.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.033+0000 7fc8b77fe700 1 -- 172.21.15.79:0/777896302 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc8c001c430 con 0x7fc8d00fbb40 2024-01-19T01:20:25.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.033+0000 7fc8b77fe700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc8b8068f60 0x7fc8b806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:25.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.033+0000 7fc8ce7fc700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc8b8068f60 0x7fc8b806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:25.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.033+0000 7fc8b77fe700 1 -- 172.21.15.79:0/777896302 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc8c0087780 con 0x7fc8d00fbb40 2024-01-19T01:20:25.037 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.034+0000 7fc8ce7fc700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc8b8068f60 0x7fc8b806b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009fd0 tx=0x7fc8c400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:25.038 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.037+0000 7fc8b77fe700 1 -- 172.21.15.79:0/777896302 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc8c0057b70 con 0x7fc8d00fbb40 2024-01-19T01:20:25.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.227+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc8d00fb2c0 con 0x7fc8d00fbb40 2024-01-19T01:20:25.228 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.227+0000 7fc8b77fe700 1 -- 172.21.15.79:0/777896302 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc8c0057700 con 0x7fc8d00fbb40 2024-01-19T01:20:25.228 INFO:teuthology.orchestra.run.smithi079.stdout:60129542176 2024-01-19T01:20:25.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc8b8068f60 msgr2=0x7fc8b806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:25.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc8b8068f60 0x7fc8b806b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009fd0 tx=0x7fc8c400b040 comp rx=0 tx=0).stop 2024-01-19T01:20:25.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d00fbb40 msgr2=0x7fc8d01136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:25.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d00fbb40 0x7fc8d01136b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc8c000b3b0 tx=0x7fc8c0009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:25.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 shutdown_connections 2024-01-19T01:20:25.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc8b8068f60 0x7fc8b806b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:25.230 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc8d00fbb40 0x7fc8d01136b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:25.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 --2- 172.21.15.79:0/777896302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8d0101b00 0x7fc8d0113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:25.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 >> 172.21.15.79:0/777896302 conn(0x7fc8d00f76c0 msgr2=0x7fc8d0065d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:25.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 shutdown_connections 2024-01-19T01:20:25.231 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:25.229+0000 7fc8d5937700 1 -- 172.21.15.79:0/777896302 wait complete. 2024-01-19T01:20:25.547 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724551 got 167503724554 for osd.5 2024-01-19T01:20:25.548 DEBUG:teuthology.parallel:result is None 2024-01-19T01:20:25.775 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:25 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/777896302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-19T01:20:25.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:25 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/777896302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-19T01:20:26.188 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:26.686 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888077 got 146028888079 for osd.4 2024-01-19T01:20:26.686 DEBUG:teuthology.parallel:result is None 2024-01-19T01:20:26.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:26 smithi152 ceph-mon[35694]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:26.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:26 smithi079 ceph-mon[28374]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:27.640 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-01-19T01:20:27.640 DEBUG:teuthology.parallel:result is None 2024-01-19T01:20:28.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2259871615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7840fbb40 msgr2=0x7fc7840fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2259871615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7840fbb40 0x7fc7840fbfb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc774004940 tx=0x7fc774009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:28.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2259871615 shutdown_connections 2024-01-19T01:20:28.211 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2259871615 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7840fbb40 0x7fc7840fbfb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2259871615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc784101b00 0x7fc784101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2259871615 >> 172.21.15.79:0/2259871615 conn(0x7fc7840719b0 msgr2=0x7fc784071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2259871615 shutdown_connections 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2259871615 wait complete. 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.176+0000 7fc78b8f5700 1 Processor -- start 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc78b8f5700 1 -- start start 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc78b8f5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7840fbb40 0x7fc7841103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:28.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc78b8f5700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc784101b00 0x7fc784110930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc78b8f5700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78410b2f0 con 0x7fc784101b00 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc78b8f5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78410b460 con 0x7fc7840fbb40 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc788e90700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc784101b00 0x7fc784110930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc788e90700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc784101b00 0x7fc784110930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51456/0 (socket says 172.21.15.79:51456) 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc788e90700 1 -- 172.21.15.79:0/2610120051 learned_addr learned my addr 172.21.15.79:0/2610120051 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc789691700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7840fbb40 0x7fc7841103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc788e90700 1 -- 172.21.15.79:0/2610120051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7840fbb40 msgr2=0x7fc7841103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc788e90700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7840fbb40 0x7fc7841103b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.213 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc788e90700 1 -- 172.21.15.79:0/2610120051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774005040 con 0x7fc784101b00 2024-01-19T01:20:28.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc788e90700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc784101b00 0x7fc784110930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc774009180 tx=0x7fc774009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:28.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.177+0000 7fc779ffb700 1 -- 172.21.15.79:0/2610120051 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc77400e040 con 0x7fc784101b00 2024-01-19T01:20:28.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.178+0000 7fc779ffb700 1 -- 172.21.15.79:0/2610120051 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc774004290 con 0x7fc784101b00 2024-01-19T01:20:28.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.178+0000 7fc779ffb700 1 -- 172.21.15.79:0/2610120051 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc77401c3f0 con 0x7fc784101b00 2024-01-19T01:20:28.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.178+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78410b6e0 con 0x7fc784101b00 2024-01-19T01:20:28.214 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.178+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78410bc20 con 0x7fc784101b00 2024-01-19T01:20:28.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.179+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7840734d0 con 0x7fc784101b00 2024-01-19T01:20:28.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.179+0000 7fc779ffb700 1 -- 172.21.15.79:0/2610120051 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc774013420 con 0x7fc784101b00 2024-01-19T01:20:28.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.180+0000 7fc779ffb700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc770068f60 0x7fc77006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:28.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.180+0000 7fc789691700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc770068f60 0x7fc77006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:28.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.180+0000 7fc779ffb700 1 -- 172.21.15.79:0/2610120051 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc774087d60 con 0x7fc784101b00 2024-01-19T01:20:28.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.180+0000 7fc789691700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc770068f60 0x7fc77006b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc780009bf0 tx=0x7fc7800091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:28.215 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.184+0000 7fc779ffb700 1 -- 172.21.15.79:0/2610120051 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc774058000 con 0x7fc784101b00 2024-01-19T01:20:28.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/2238911475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 msgr2=0x7f9dc40fdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/2238911475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fdfd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0004940 tx=0x7f9dc0009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/2238911475 shutdown_connections 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/2238911475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dc40fe5a0 0x7f9dc4106be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/2238911475 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fdfd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/2238911475 >> 172.21.15.79:0/2238911475 conn(0x7f9dc40f7690 msgr2=0x7f9dc40f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/2238911475 shutdown_connections 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/2238911475 wait complete. 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 Processor -- start 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.086+0000 7f9dcc3dd700 1 -- start start 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dcc3dd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:28.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dcc3dd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dc40fe5a0 0x7f9dc40fbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:28.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dcc3dd700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40fc570 con 0x7f9dc40fdc00 2024-01-19T01:20:28.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dcc3dd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40fc6e0 con 0x7f9dc40fe5a0 2024-01-19T01:20:28.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dca179700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:28.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dca179700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51426/0 (socket says 172.21.15.79:51426) 2024-01-19T01:20:28.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dca179700 1 -- 172.21.15.79:0/3983705803 learned_addr learned my addr 172.21.15.79:0/3983705803 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:28.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dc9978700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dc40fe5a0 0x7f9dc40fbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:28.315 INFO:teuthology.orchestra.run.smithi079.stdout:81604378651 2024-01-19T01:20:28.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dca179700 1 -- 172.21.15.79:0/3983705803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dc40fe5a0 msgr2=0x7f9dc40fbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dca179700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dc40fe5a0 0x7f9dc40fbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dca179700 1 -- 172.21.15.79:0/3983705803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0005040 con 0x7f9dc40fdc00 2024-01-19T01:20:28.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dca179700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fba30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000b3b0 tx=0x7f9dc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:28.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dbaffd700 1 -- 172.21.15.79:0/3983705803 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc000e040 con 0x7f9dc40fdc00 2024-01-19T01:20:28.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4119870 con 0x7f9dc40fdc00 2024-01-19T01:20:28.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dbaffd700 1 -- 172.21.15.79:0/3983705803 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dc0004290 con 0x7f9dc40fdc00 2024-01-19T01:20:28.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dbaffd700 1 -- 172.21.15.79:0/3983705803 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc001c3f0 con 0x7f9dc40fdc00 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.087+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4119cd0 con 0x7f9dc40fdc00 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.088+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc41041f0 con 0x7f9dc40fdc00 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.089+0000 7f9dbaffd700 1 -- 172.21.15.79:0/3983705803 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9dc000bc50 con 0x7f9dc40fdc00 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.089+0000 7f9dbaffd700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9db0068f60 0x7f9db006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.089+0000 7f9dbaffd700 1 -- 172.21.15.79:0/3983705803 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9dc0087ef0 con 0x7f9dc40fdc00 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.089+0000 7f9dc9978700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9db0068f60 0x7f9db006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.090+0000 7f9dc9978700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9db0068f60 0x7f9db006b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9db4007fd0 tx=0x7f9db4009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.093+0000 7f9dbaffd700 1 -- 172.21.15.79:0/3983705803 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9dc0054700 con 0x7f9dc40fdc00 2024-01-19T01:20:28.317 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.297+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9dc404bb80 con 0x7f9dc40fdc00 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.297+0000 7f9dbaffd700 1 -- 172.21.15.79:0/3983705803 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9dc0057d20 con 0x7f9dc40fdc00 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9db0068f60 msgr2=0x7f9db006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9db0068f60 0x7f9db006b410 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9db4007fd0 tx=0x7f9db4009040 comp rx=0 tx=0).stop 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 msgr2=0x7f9dc40fba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fba30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9dc000b3b0 tx=0x7f9dc0009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 shutdown_connections 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9db0068f60 0x7f9db006b410 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9dc40fdc00 0x7f9dc40fba30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 --2- 172.21.15.79:0/3983705803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9dc40fe5a0 0x7f9dc40fbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 >> 172.21.15.79:0/3983705803 conn(0x7f9dc40f7690 msgr2=0x7f9dc4101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:28.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 shutdown_connections 2024-01-19T01:20:28.319 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.299+0000 7f9dcc3dd700 1 -- 172.21.15.79:0/3983705803 wait complete. 2024-01-19T01:20:28.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.386+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc78404bb80 con 0x7fc784101b00 2024-01-19T01:20:28.388 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.386+0000 7fc779ffb700 1 -- 172.21.15.79:0/2610120051 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc774057b90 con 0x7fc784101b00 2024-01-19T01:20:28.389 INFO:teuthology.orchestra.run.smithi079.stdout:38654705702 2024-01-19T01:20:28.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.388+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc770068f60 msgr2=0x7fc77006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.388+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc770068f60 0x7fc77006b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc780009bf0 tx=0x7fc7800091a0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.388+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc784101b00 msgr2=0x7fc784110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:28.390 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.388+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc784101b00 0x7fc784110930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc774009180 tx=0x7fc774009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:28.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.389+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 shutdown_connections 2024-01-19T01:20:28.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.389+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc770068f60 0x7fc77006b410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.389+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7840fbb40 0x7fc7841103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.389+0000 7fc78b8f5700 1 --2- 172.21.15.79:0/2610120051 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc784101b00 0x7fc784110930 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:28.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.389+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 >> 172.21.15.79:0/2610120051 conn(0x7fc7840719b0 msgr2=0x7fc7840654b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:28.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.389+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 shutdown_connections 2024-01-19T01:20:28.391 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:28.389+0000 7fc78b8f5700 1 -- 172.21.15.79:0/2610120051 wait complete. 2024-01-19T01:20:28.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:28 smithi079 ceph-mon[28374]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:28.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:28 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3983705803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-19T01:20:28.825 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:28 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2610120051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-19T01:20:28.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:28 smithi152 ceph-mon[35694]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:28.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:28 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3983705803' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-19T01:20:28.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:28 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2610120051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-19T01:20:29.134 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-01-19T01:20:29.134 DEBUG:teuthology.parallel:result is None 2024-01-19T01:20:30.798 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378651 for osd.2 2024-01-19T01:20:30.798 DEBUG:teuthology.parallel:result is None 2024-01-19T01:20:30.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:30 smithi152 ceph-mon[35694]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:30.926 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:30 smithi079 ceph-mon[28374]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:31.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.272+0000 7f255ab5d700 1 -- 172.21.15.79:0/2129694826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 msgr2=0x7f25540fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:31.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.272+0000 7f255ab5d700 1 --2- 172.21.15.79:0/2129694826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fb630 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2540004940 tx=0x7f2540009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:31.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.272+0000 7f255ab5d700 1 -- 172.21.15.79:0/2129694826 shutdown_connections 2024-01-19T01:20:31.275 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.272+0000 7f255ab5d700 1 --2- 172.21.15.79:0/2129694826 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fb630 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.272+0000 7f255ab5d700 1 --2- 172.21.15.79:0/2129694826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2554102c70 0x7f2554103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.272+0000 7f255ab5d700 1 -- 172.21.15.79:0/2129694826 >> 172.21.15.79:0/2129694826 conn(0x7f25540711a0 msgr2=0x7f25540715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.272+0000 7f255ab5d700 1 -- 172.21.15.79:0/2129694826 shutdown_connections 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.273+0000 7f255ab5d700 1 -- 172.21.15.79:0/2129694826 wait complete. 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.273+0000 7f255ab5d700 1 Processor -- start 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.273+0000 7f255ab5d700 1 -- start start 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.273+0000 7f255ab5d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.273+0000 7f255ab5d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2554102c70 0x7f25540fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.273+0000 7f255ab5d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2554111540 con 0x7f25540fb1e0 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.273+0000 7f255ab5d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25540fce40 con 0x7f2554102c70 2024-01-19T01:20:31.276 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25588f9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25588f9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53954/0 (socket says 172.21.15.79:53954) 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25588f9700 1 -- 172.21.15.79:0/3480581392 learned_addr learned my addr 172.21.15.79:0/3480581392 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f2553fff700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2554102c70 0x7f25540fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25588f9700 1 -- 172.21.15.79:0/3480581392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2554102c70 msgr2=0x7f25540fc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25588f9700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2554102c70 0x7f25540fc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25588f9700 1 -- 172.21.15.79:0/3480581392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2540005040 con 0x7f25540fb1e0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25588f9700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fc3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f254800b790 tx=0x7f2548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25517fa700 1 -- 172.21.15.79:0/3480581392 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f254800f6f0 con 0x7f25540fb1e0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25540fd120 con 0x7f25540fb1e0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25517fa700 1 -- 172.21.15.79:0/3480581392 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f254800fd00 con 0x7f25540fb1e0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f25517fa700 1 -- 172.21.15.79:0/3480581392 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25480103a0 con 0x7f25540fb1e0 2024-01-19T01:20:31.277 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.274+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255411e1b0 con 0x7f25540fb1e0 2024-01-19T01:20:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.275+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2554072cc0 con 0x7f25540fb1e0 2024-01-19T01:20:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.276+0000 7f25517fa700 1 -- 172.21.15.79:0/3480581392 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f254800f850 con 0x7f25540fb1e0 2024-01-19T01:20:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.276+0000 7f25517fa700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2544068f60 0x7f254406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.276+0000 7f2553fff700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2544068f60 0x7f254406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:31.278 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.276+0000 7f25517fa700 1 -- 172.21.15.79:0/3480581392 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2548086870 con 0x7f25540fb1e0 2024-01-19T01:20:31.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.277+0000 7f2553fff700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2544068f60 0x7f254406b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f25540fe4d0 tx=0x7f2540009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:31.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.281+0000 7f25517fa700 1 -- 172.21.15.79:0/3480581392 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2548053080 con 0x7f25540fb1e0 2024-01-19T01:20:31.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.465+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2554109bf0 con 0x7f25540fb1e0 2024-01-19T01:20:31.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.465+0000 7f25517fa700 1 -- 172.21.15.79:0/3480581392 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f25480566a0 con 0x7f25540fb1e0 2024-01-19T01:20:31.467 INFO:teuthology.orchestra.run.smithi079.stdout:111669149719 2024-01-19T01:20:31.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2544068f60 msgr2=0x7f254406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:31.469 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2544068f60 0x7f254406b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f25540fe4d0 tx=0x7f2540009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:31.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 msgr2=0x7f25540fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:31.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fc3c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f254800b790 tx=0x7f2548009250 comp rx=0 tx=0).stop 2024-01-19T01:20:31.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 shutdown_connections 2024-01-19T01:20:31.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2544068f60 0x7f254406b410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:31.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f25540fb1e0 0x7f25540fc3c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:31.470 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 --2- 172.21.15.79:0/3480581392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2554102c70 0x7f25540fc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:31.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 >> 172.21.15.79:0/3480581392 conn(0x7f25540711a0 msgr2=0x7f25540fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:31.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 shutdown_connections 2024-01-19T01:20:31.471 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:31.468+0000 7f255ab5d700 1 -- 172.21.15.79:0/3480581392 wait complete. 2024-01-19T01:20:31.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:31 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3480581392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-19T01:20:32.000 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:31 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3480581392' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-19T01:20:32.128 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149717 got 111669149719 for osd.3 2024-01-19T01:20:32.128 DEBUG:teuthology.parallel:result is None 2024-01-19T01:20:32.129 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-01-19T01:20:32.129 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph pg dump --format=json 2024-01-19T01:20:32.392 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:32.778 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:32 smithi079 ceph-mon[28374]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:32.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:32 smithi152 ceph-mon[35694]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 -- 172.21.15.79:0/2524435577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 msgr2=0x7f5044101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 --2- 172.21.15.79:0/2524435577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f5044101ed0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f503c004940 tx=0x7f503c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 -- 172.21.15.79:0/2524435577 shutdown_connections 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 --2- 172.21.15.79:0/2524435577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50440fbb40 0x7f50440fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 --2- 172.21.15.79:0/2524435577 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f5044101ed0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 -- 172.21.15.79:0/2524435577 >> 172.21.15.79:0/2524435577 conn(0x7f50440719b0 msgr2=0x7f5044071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 -- 172.21.15.79:0/2524435577 shutdown_connections 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.156+0000 7f504c515700 1 -- 172.21.15.79:0/2524435577 wait complete. 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504c515700 1 Processor -- start 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504c515700 1 -- start start 2024-01-19T01:20:34.159 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504c515700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50440fbb40 0x7f504410c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504c515700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f504410c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504c515700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504410ce40 con 0x7f5044101b00 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504c515700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504410cfb0 con 0x7f50440fbb40 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f5049ab0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f504410c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f5049ab0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f504410c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53966/0 (socket says 172.21.15.79:53966) 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f5049ab0700 1 -- 172.21.15.79:0/2069393390 learned_addr learned my addr 172.21.15.79:0/2069393390 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504a2b1700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50440fbb40 0x7f504410c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f5049ab0700 1 -- 172.21.15.79:0/2069393390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50440fbb40 msgr2=0x7f504410c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f5049ab0700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50440fbb40 0x7f504410c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f5049ab0700 1 -- 172.21.15.79:0/2069393390 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c005040 con 0x7f5044101b00 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f504a2b1700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50440fbb40 0x7f504410c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.157+0000 7f5049ab0700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f504410c870 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f503400b790 tx=0x7f5034009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.158+0000 7f503affd700 1 -- 172.21.15.79:0/2069393390 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f503400f770 con 0x7f5044101b00 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.158+0000 7f503affd700 1 -- 172.21.15.79:0/2069393390 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f503400fd80 con 0x7f5044101b00 2024-01-19T01:20:34.160 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.158+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50441199a0 con 0x7f5044101b00 2024-01-19T01:20:34.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.158+0000 7f503affd700 1 -- 172.21.15.79:0/2069393390 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f503400d310 con 0x7f5044101b00 2024-01-19T01:20:34.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.158+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5044119e30 con 0x7f5044101b00 2024-01-19T01:20:34.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.159+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5044103fb0 con 0x7f5044101b00 2024-01-19T01:20:34.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.159+0000 7f503affd700 1 -- 172.21.15.79:0/2069393390 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5034010420 con 0x7f5044101b00 2024-01-19T01:20:34.161 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.159+0000 7f503affd700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5030068f60 0x7f503006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:34.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.160+0000 7f504a2b1700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5030068f60 0x7f503006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:34.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.160+0000 7f503affd700 1 -- 172.21.15.79:0/2069393390 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5034086950 con 0x7f5044101b00 2024-01-19T01:20:34.162 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.160+0000 7f504a2b1700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5030068f60 0x7f503006b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f503c004f90 tx=0x7f503c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:34.164 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.164+0000 7f503affd700 1 -- 172.21.15.79:0/2069393390 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5034056c90 con 0x7f5044101b00 2024-01-19T01:20:34.363 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.362+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5044065b70 con 0x7f5030068f60 2024-01-19T01:20:34.364 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.363+0000 7f503affd700 1 -- 172.21.15.79:0/2069393390 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28350 (secure 0 0 0) 0x7f5044065b70 con 0x7f5030068f60 2024-01-19T01:20:34.364 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.365+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5030068f60 msgr2=0x7f503006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.365+0000 7f504c515700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5030068f60 0x7f503006b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f503c004f90 tx=0x7f503c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.365+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 msgr2=0x7f504410c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.365+0000 7f504c515700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f504410c870 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f503400b790 tx=0x7f5034009250 comp rx=0 tx=0).stop 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.366+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 shutdown_connections 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.366+0000 7f504c515700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f5030068f60 0x7f503006b410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.366+0000 7f504c515700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50440fbb40 0x7f504410c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.366+0000 7f504c515700 1 --2- 172.21.15.79:0/2069393390 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f5044101b00 0x7f504410c870 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.366+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 >> 172.21.15.79:0/2069393390 conn(0x7f50440719b0 msgr2=0x7f50440faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.366+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 shutdown_connections 2024-01-19T01:20:34.367 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:34.366+0000 7f504c515700 1 -- 172.21.15.79:0/2069393390 wait complete. 2024-01-19T01:20:34.368 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-01-19T01:20:34.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:34 smithi152 ceph-mon[35694]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:34.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:34.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:34 smithi079 ceph-mon[28374]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:34.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:35.049 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-01-19T01:20:33.208895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001991"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-19T01:19:43.534613+0000","last_change":"2024-01-19T01:19:18.623600+0000","last_active":"2024-01-19T01:19:43.534613+0000","last_peered":"2024-01-19T01:19:43.534613+0000","last_clean":"2024-01-19T01:19:43.534613+0000","last_became_active":"2024-01-19T01:19:18.623298+0000","last_became_peered":"2024-01-19T01:19:18.623298+0000","last_unstale":"2024-01-19T01:19:43.534613+0000","last_undegraded":"2024-01-19T01:19:43.534613+0000","last_fullsized":"2024-01-19T01:19:43.534613+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T01:18:29.955211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T01:18:29.955211+0000","last_clean_scrub_stamp":"2024-01-19T01:18:29.955211+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-20T09:14:52.605120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.436}]},{"osd":1,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56699999999999995}]},{"osd":3,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.67800000000000005}]},{"osd":1,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.59899999999999998}]},{"osd":2,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Jan 19 01:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.28999999999999998}]},{"osd":1,"last update":"Fri Jan 19 01:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54600000000000004}]},{"osd":4,"last update":"Fri Jan 19 01:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 19 01:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.312},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51400000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.33300000000000002,"15min":0.32700000000000001},"min":{"1min":0.23100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.26200000000000001}]},{"osd":2,"last update":"Fri Jan 19 01:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Jan 19 01:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:20:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.29999999999999999,"15min":0.29399999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.308,"15min":0.29799999999999999},"min":{"1min":0.25600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.26300000000000001}]},{"osd":2,"last update":"Fri Jan 19 01:19:33 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.251}]},{"osd":3,"last update":"Fri Jan 19 01:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Jan 19 01:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-19T01:20:35.050 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph pg dump --format=json 2024-01-19T01:20:35.306 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:35.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:35 smithi079 ceph-mon[28374]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T01:20:35.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:35 smithi152 ceph-mon[35694]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T01:20:36.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:36 smithi079 ceph-mon[28374]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:36.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:36 smithi152 ceph-mon[35694]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:37.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 -- 172.21.15.79:0/3306841155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c0fb1e0 msgr2=0x7f3f7c0fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:37.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 --2- 172.21.15.79:0/3306841155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c0fb1e0 0x7f3f7c0fb630 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c004940 tx=0x7f3f6c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:37.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 -- 172.21.15.79:0/3306841155 shutdown_connections 2024-01-19T01:20:37.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 --2- 172.21.15.79:0/3306841155 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c0fb1e0 0x7f3f7c0fb630 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:37.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 --2- 172.21.15.79:0/3306841155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c102c70 0x7f3f7c103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:37.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 -- 172.21.15.79:0/3306841155 >> 172.21.15.79:0/3306841155 conn(0x7f3f7c0711a0 msgr2=0x7f3f7c0715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:37.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 -- 172.21.15.79:0/3306841155 shutdown_connections 2024-01-19T01:20:37.119 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.117+0000 7f3f8344d700 1 -- 172.21.15.79:0/3306841155 wait complete. 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f8344d700 1 Processor -- start 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f8344d700 1 -- start start 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f8344d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c0fb1e0 0x7f3f7c113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f8344d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c102c70 0x7f3f7c113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f8344d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c114200 con 0x7f3f7c102c70 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f8344d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c10d4f0 con 0x7f3f7c0fb1e0 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f809e8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c102c70 0x7f3f7c113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f809e8700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c102c70 0x7f3f7c113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53994/0 (socket says 172.21.15.79:53994) 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f809e8700 1 -- 172.21.15.79:0/2752828501 learned_addr learned my addr 172.21.15.79:0/2752828501 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f811e9700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c0fb1e0 0x7f3f7c113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f809e8700 1 -- 172.21.15.79:0/2752828501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c0fb1e0 msgr2=0x7f3f7c113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f809e8700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c0fb1e0 0x7f3f7c113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:37.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.118+0000 7f3f809e8700 1 -- 172.21.15.79:0/2752828501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c005040 con 0x7f3f7c102c70 2024-01-19T01:20:37.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.119+0000 7f3f809e8700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c102c70 0x7f3f7c113bb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00b3b0 tx=0x7f3f6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:37.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.119+0000 7f3f71ffb700 1 -- 172.21.15.79:0/2752828501 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f6c00e040 con 0x7f3f7c102c70 2024-01-19T01:20:37.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.119+0000 7f3f71ffb700 1 -- 172.21.15.79:0/2752828501 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f6c01c430 con 0x7f3f7c102c70 2024-01-19T01:20:37.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.119+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c10d720 con 0x7f3f7c102c70 2024-01-19T01:20:37.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.119+0000 7f3f71ffb700 1 -- 172.21.15.79:0/2752828501 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f6c013640 con 0x7f3f7c102c70 2024-01-19T01:20:37.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.119+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c10dc60 con 0x7f3f7c102c70 2024-01-19T01:20:37.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.120+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c105050 con 0x7f3f7c102c70 2024-01-19T01:20:37.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.121+0000 7f3f71ffb700 1 -- 172.21.15.79:0/2752828501 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f3f6c01c5a0 con 0x7f3f7c102c70 2024-01-19T01:20:37.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.121+0000 7f3f71ffb700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3f68068f60 0x7f3f6806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:37.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.121+0000 7f3f71ffb700 1 -- 172.21.15.79:0/2752828501 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3f6c087480 con 0x7f3f7c102c70 2024-01-19T01:20:37.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.121+0000 7f3f811e9700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3f68068f60 0x7f3f6806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:37.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.121+0000 7f3f811e9700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3f68068f60 0x7f3f6806b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f74004550 tx=0x7f3f74007ed0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:37.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.125+0000 7f3f71ffb700 1 -- 172.21.15.79:0/2752828501 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3f6c057870 con 0x7f3f7c102c70 2024-01-19T01:20:37.304 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.303+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f7c04bbf0 con 0x7f3f68068f60 2024-01-19T01:20:37.305 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.304+0000 7f3f71ffb700 1 -- 172.21.15.79:0/2752828501 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28294 (secure 0 0 0) 0x7f3f7c04bbf0 con 0x7f3f68068f60 2024-01-19T01:20:37.306 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:20:37.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3f68068f60 msgr2=0x7f3f6806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:37.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3f68068f60 0x7f3f6806b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f74004550 tx=0x7f3f74007ed0 comp rx=0 tx=0).stop 2024-01-19T01:20:37.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c102c70 msgr2=0x7f3f7c113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:37.308 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c102c70 0x7f3f7c113bb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00b3b0 tx=0x7f3f6c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:37.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 shutdown_connections 2024-01-19T01:20:37.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3f68068f60 0x7f3f6806b410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:37.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f7c0fb1e0 0x7f3f7c113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:37.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 --2- 172.21.15.79:0/2752828501 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f3f7c102c70 0x7f3f7c113bb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:37.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 >> 172.21.15.79:0/2752828501 conn(0x7f3f7c0711a0 msgr2=0x7f3f7c0fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:37.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 shutdown_connections 2024-01-19T01:20:37.309 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:37.307+0000 7f3f8344d700 1 -- 172.21.15.79:0/2752828501 wait complete. 2024-01-19T01:20:37.310 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-01-19T01:20:38.011 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-01-19T01:20:37.209525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787600,"kb_used_data":3872,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683312,"statfs":{"total":575970213888,"available":574139711488,"internally_reserved":0,"allocated":3964928,"data_stored":2605939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002013"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-19T01:19:43.534613+0000","last_change":"2024-01-19T01:19:18.623600+0000","last_active":"2024-01-19T01:19:43.534613+0000","last_peered":"2024-01-19T01:19:43.534613+0000","last_clean":"2024-01-19T01:19:43.534613+0000","last_became_active":"2024-01-19T01:19:18.623298+0000","last_became_peered":"2024-01-19T01:19:18.623298+0000","last_unstale":"2024-01-19T01:19:43.534613+0000","last_undegraded":"2024-01-19T01:19:43.534613+0000","last_fullsized":"2024-01-19T01:19:43.534613+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T01:18:29.955211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T01:18:29.955211+0000","last_clean_scrub_stamp":"2024-01-19T01:18:29.955211+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-20T09:14:52.605120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.56200000000000006}]},{"osd":1,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.54400000000000004}]},{"osd":2,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.499}]},{"osd":3,"last update":"Fri Jan 19 01:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298220,"kb_used_data":932,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446932,"statfs":{"total":95995035648,"available":95689658368,"internally_reserved":0,"allocated":954368,"data_stored":729238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.45200000000000001}]},{"osd":2,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Jan 19 01:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:20:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.34899999999999998,"15min":0.35199999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.379,"15min":0.38700000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.28799999999999998}]},{"osd":1,"last update":"Fri Jan 19 01:20:31 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.379,"15min":0.38700000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.253},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.39500000000000002,"15min":0.40500000000000003},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Jan 19 01:20:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.42499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jan 19 01:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.312},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51400000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.33300000000000002,"15min":0.32700000000000001},"min":{"1min":0.23100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.26200000000000001}]},{"osd":2,"last update":"Fri Jan 19 01:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Jan 19 01:19:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Jan 19 01:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jan 19 01:20:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.29999999999999999,"15min":0.29399999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.308,"15min":0.29799999999999999},"min":{"1min":0.25600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.26300000000000001}]},{"osd":2,"last update":"Fri Jan 19 01:19:33 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.251}]},{"osd":3,"last update":"Fri Jan 19 01:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Jan 19 01:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.47099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-19T01:20:38.012 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-01-19T01:20:38.012 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-19T01:20:38.012 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-01-19T01:20:38.012 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-19T01:20:38.270 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:38.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:38 smithi152 ceph-mon[35694]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:38.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:38 smithi152 ceph-mon[35694]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T01:20:38.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:38 smithi079 ceph-mon[28374]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:38.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:38 smithi079 ceph-mon[28374]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-19T01:20:39.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.968+0000 7f6e11e21700 1 -- 172.21.15.79:0/3763076211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c0fb1e0 msgr2=0x7f6e0c0fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:39.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.968+0000 7f6e11e21700 1 --2- 172.21.15.79:0/3763076211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c0fb1e0 0x7f6e0c0fb630 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6e00004940 tx=0x7f6e00009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:39.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.968+0000 7f6e11e21700 1 -- 172.21.15.79:0/3763076211 shutdown_connections 2024-01-19T01:20:39.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.968+0000 7f6e11e21700 1 --2- 172.21.15.79:0/3763076211 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c0fb1e0 0x7f6e0c0fb630 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:39.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.968+0000 7f6e11e21700 1 --2- 172.21.15.79:0/3763076211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c102c70 0x7f6e0c103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:39.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.968+0000 7f6e11e21700 1 -- 172.21.15.79:0/3763076211 >> 172.21.15.79:0/3763076211 conn(0x7f6e0c0711a0 msgr2=0x7f6e0c0715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:39.971 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 -- 172.21.15.79:0/3763076211 shutdown_connections 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 -- 172.21.15.79:0/3763076211 wait complete. 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 Processor -- start 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 -- start start 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0fb1e0 0x7f6e0c113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c102c70 0x7f6e0c113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c1141d0 con 0x7f6e0c102c70 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e11e21700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c10d4c0 con 0x7f6e0c0fb1e0 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.969+0000 7f6e0affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c102c70 0x7f6e0c113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0affd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c102c70 0x7f6e0c113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51596/0 (socket says 172.21.15.79:51596) 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0affd700 1 -- 172.21.15.79:0/1219518578 learned_addr learned my addr 172.21.15.79:0/1219518578 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0b7fe700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0fb1e0 0x7f6e0c113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0affd700 1 -- 172.21.15.79:0/1219518578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0fb1e0 msgr2=0x7f6e0c113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0affd700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0fb1e0 0x7f6e0c113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:39.972 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0affd700 1 -- 172.21.15.79:0/1219518578 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00005040 con 0x7f6e0c102c70 2024-01-19T01:20:39.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0b7fe700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0fb1e0 0x7f6e0c113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:20:39.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e0affd700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c102c70 0x7f6e0c113b80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b3b0 tx=0x7f6e00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:39.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6df3fff700 1 -- 172.21.15.79:0/1219518578 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e0000e040 con 0x7f6e0c102c70 2024-01-19T01:20:39.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6df3fff700 1 -- 172.21.15.79:0/1219518578 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e0000bc50 con 0x7f6e0c102c70 2024-01-19T01:20:39.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c10d6f0 con 0x7f6e0c102c70 2024-01-19T01:20:39.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6df3fff700 1 -- 172.21.15.79:0/1219518578 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e00013420 con 0x7f6e0c102c70 2024-01-19T01:20:39.973 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.970+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c10dc30 con 0x7f6e0c102c70 2024-01-19T01:20:39.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.971+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c105050 con 0x7f6e0c102c70 2024-01-19T01:20:39.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.972+0000 7f6df3fff700 1 -- 172.21.15.79:0/1219518578 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6e0001c430 con 0x7f6e0c102c70 2024-01-19T01:20:39.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.972+0000 7f6df3fff700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6df4068f60 0x7f6df406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:39.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.972+0000 7f6df3fff700 1 -- 172.21.15.79:0/1219518578 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6e00087830 con 0x7f6e0c102c70 2024-01-19T01:20:39.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.972+0000 7f6e0b7fe700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6df4068f60 0x7f6df406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:39.974 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.973+0000 7f6e0b7fe700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6df4068f60 0x7f6df406b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0041d0 tx=0x7f6dfc009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:39.977 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:39.976+0000 7f6df3fff700 1 -- 172.21.15.79:0/1219518578 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6e00057b80 con 0x7f6e0c102c70 2024-01-19T01:20:40.201 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.200+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6e0c10ebe0 con 0x7f6e0c102c70 2024-01-19T01:20:40.202 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.201+0000 7f6df3fff700 1 -- 172.21.15.79:0/1219518578 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6e00057710 con 0x7f6e0c102c70 2024-01-19T01:20:40.202 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:20:40.202 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6df4068f60 msgr2=0x7f6df406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6df4068f60 0x7f6df406b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0041d0 tx=0x7f6dfc009040 comp rx=0 tx=0).stop 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c102c70 msgr2=0x7f6e0c113b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c102c70 0x7f6e0c113b80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b3b0 tx=0x7f6e00009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 shutdown_connections 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6df4068f60 0x7f6df406b410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e0c0fb1e0 0x7f6e0c113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 --2- 172.21.15.79:0/1219518578 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6e0c102c70 0x7f6e0c113b80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:40.204 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 >> 172.21.15.79:0/1219518578 conn(0x7f6e0c0711a0 msgr2=0x7f6e0c0fa3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:40.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 shutdown_connections 2024-01-19T01:20:40.205 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:40.203+0000 7f6e11e21700 1 -- 172.21.15.79:0/1219518578 wait complete. 2024-01-19T01:20:40.895 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:40 smithi079 ceph-mon[28374]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:40.895 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:40 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1219518578' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-19T01:20:40.923 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-01-19T01:20:40.923 INFO:tasks.cephadm:Setup complete, yielding 2024-01-19T01:20:40.923 INFO:teuthology.run_tasks:Running task print... 2024-01-19T01:20:40.932 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-01-19T01:20:40.932 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T01:20:40.941 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:20:40.941 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-01-19T01:20:40.942 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:40 smithi152 ceph-mon[35694]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:40.942 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:40 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1219518578' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-19T01:20:41.200 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:42.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 -- 172.21.15.79:0/2648244000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fcbf0 msgr2=0x7f908c0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:42.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 --2- 172.21.15.79:0/2648244000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fcbf0 0x7f908c0fd000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9080004940 tx=0x7f9080009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:42.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 -- 172.21.15.79:0/2648244000 shutdown_connections 2024-01-19T01:20:42.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 --2- 172.21.15.79:0/2648244000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f908c0fde00 0x7f908c0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:42.759 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 --2- 172.21.15.79:0/2648244000 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fcbf0 0x7f908c0fd000 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 -- 172.21.15.79:0/2648244000 >> 172.21.15.79:0/2648244000 conn(0x7f908c0f82d0 msgr2=0x7f908c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 -- 172.21.15.79:0/2648244000 shutdown_connections 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 -- 172.21.15.79:0/2648244000 wait complete. 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 Processor -- start 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.757+0000 7f909307b700 1 -- start start 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f909307b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f908c0fcbf0 0x7f908c071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f909307b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fde00 0x7f908c06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f909307b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c070620 con 0x7f908c0fde00 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f909307b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c070790 con 0x7f908c0fcbf0 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f908bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fde00 0x7f908c06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f908bfff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fde00 0x7f908c06ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51606/0 (socket says 172.21.15.79:51606) 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f908bfff700 1 -- 172.21.15.79:0/3300647456 learned_addr learned my addr 172.21.15.79:0/3300647456 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f9090e17700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f908c0fcbf0 0x7f908c071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f908bfff700 1 -- 172.21.15.79:0/3300647456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f908c0fcbf0 msgr2=0x7f908c071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:42.760 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f908bfff700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f908c0fcbf0 0x7f908c071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:42.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f908bfff700 1 -- 172.21.15.79:0/3300647456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9080005040 con 0x7f908c0fde00 2024-01-19T01:20:42.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f908bfff700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fde00 0x7f908c06ff90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f907800b790 tx=0x7f9078009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:42.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f90897fa700 1 -- 172.21.15.79:0/3300647456 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f907800f770 con 0x7f908c0fde00 2024-01-19T01:20:42.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f90897fa700 1 -- 172.21.15.79:0/3300647456 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f907800fd80 con 0x7f908c0fde00 2024-01-19T01:20:42.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c070a70 con 0x7f908c0fde00 2024-01-19T01:20:42.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f90897fa700 1 -- 172.21.15.79:0/3300647456 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f907800d310 con 0x7f908c0fde00 2024-01-19T01:20:42.761 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.758+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c070df0 con 0x7f908c0fde00 2024-01-19T01:20:42.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.759+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c04bb80 con 0x7f908c0fde00 2024-01-19T01:20:42.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.760+0000 7f90897fa700 1 -- 172.21.15.79:0/3300647456 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9078010420 con 0x7f908c0fde00 2024-01-19T01:20:42.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.760+0000 7f90897fa700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f907c068f60 0x7f907c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:42.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.760+0000 7f90897fa700 1 -- 172.21.15.79:0/3300647456 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9078086a40 con 0x7f908c0fde00 2024-01-19T01:20:42.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.760+0000 7f9090e17700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f907c068f60 0x7f907c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:42.762 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.760+0000 7f9090e17700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f907c068f60 0x7f907c06b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f908000b3b0 tx=0x7f9080009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:42.765 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.764+0000 7f90897fa700 1 -- 172.21.15.79:0/3300647456 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9078056ce0 con 0x7f908c0fde00 2024-01-19T01:20:42.951 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.950+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f908c105610 con 0x7f908c0fde00 2024-01-19T01:20:42.951 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:42 smithi079 ceph-mon[28374]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:42.955 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.954+0000 7f90897fa700 1 -- 172.21.15.79:0/3300647456 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f9078056870 con 0x7f908c0fde00 2024-01-19T01:20:42.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f907c068f60 msgr2=0x7f907c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:42.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f907c068f60 0x7f907c06b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f908000b3b0 tx=0x7f9080009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:42.957 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fde00 msgr2=0x7f908c06ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fde00 0x7f908c06ff90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f907800b790 tx=0x7f9078009250 comp rx=0 tx=0).stop 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 shutdown_connections 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f907c068f60 0x7f907c06b410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f908c0fcbf0 0x7f908c071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 --2- 172.21.15.79:0/3300647456 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f908c0fde00 0x7f908c06ff90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 >> 172.21.15.79:0/3300647456 conn(0x7f908c0f82d0 msgr2=0x7f908c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 shutdown_connections 2024-01-19T01:20:42.958 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:42.956+0000 7f909307b700 1 -- 172.21.15.79:0/3300647456 wait complete. 2024-01-19T01:20:43.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:42 smithi152 ceph-mon[35694]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:43.548 INFO:teuthology.run_tasks:Running task print... 2024-01-19T01:20:43.557 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-01-19T01:20:43.557 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T01:20:43.566 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:20:43.566 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph orch status' 2024-01-19T01:20:43.820 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:44.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:43 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3300647456' entity='client.admin' 2024-01-19T01:20:44.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:43 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:20:44.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:43 smithi079 ceph-mon[28374]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:43 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:20:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:43 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:20:44.249 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:43 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:20:44.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:43 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3300647456' entity='client.admin' 2024-01-19T01:20:44.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:20:44.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:43 smithi152 ceph-mon[35694]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:44.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:20:44.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:20:44.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:43 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:20:45.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 -- 172.21.15.79:0/532510693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fde00 msgr2=0x7fdbc80fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:45.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/532510693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fde00 0x7fdbc80fe250 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8004940 tx=0x7fdbb8009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:45.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 -- 172.21.15.79:0/532510693 shutdown_connections 2024-01-19T01:20:45.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/532510693 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fde00 0x7fdbc80fe250 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:45.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/532510693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc80fcbf0 0x7fdbc80fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:45.502 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 -- 172.21.15.79:0/532510693 >> 172.21.15.79:0/532510693 conn(0x7fdbc80f82d0 msgr2=0x7fdbc80fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 -- 172.21.15.79:0/532510693 shutdown_connections 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.499+0000 7fdbcd47d700 1 -- 172.21.15.79:0/532510693 wait complete. 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbcd47d700 1 Processor -- start 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbcd47d700 1 -- start start 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbcd47d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fcbf0 0x7fdbc810d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbcd47d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc80fde00 0x7fdbc810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbcd47d700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc810e010 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbcd47d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc810e8b0 con 0x7fdbc80fde00 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc6ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fcbf0 0x7fdbc810d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc6ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fcbf0 0x7fdbc810d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51610/0 (socket says 172.21.15.79:51610) 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc6ffd700 1 -- 172.21.15.79:0/2626229689 learned_addr learned my addr 172.21.15.79:0/2626229689 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc67fc700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc80fde00 0x7fdbc810da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:45.503 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc6ffd700 1 -- 172.21.15.79:0/2626229689 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc80fde00 msgr2=0x7fdbc810da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc6ffd700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc80fde00 0x7fdbc810da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc6ffd700 1 -- 172.21.15.79:0/2626229689 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb8005040 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.500+0000 7fdbc6ffd700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fcbf0 0x7fdbc810d4d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc002a80 tx=0x7fdbbc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.501+0000 7fdbaf7fe700 1 -- 172.21.15.79:0/2626229689 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbbc007cc0 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.501+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc810eb90 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.501+0000 7fdbaf7fe700 1 -- 172.21.15.79:0/2626229689 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbbc00f040 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.501+0000 7fdbaf7fe700 1 -- 172.21.15.79:0/2626229689 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbbc0095d0 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.501+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc810f0d0 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.502+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc804bb80 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.502+0000 7fdbaf7fe700 1 -- 172.21.15.79:0/2626229689 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdbbc009730 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.502+0000 7fdbaf7fe700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdbb0068f60 0x7fdbb006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.503+0000 7fdbaf7fe700 1 -- 172.21.15.79:0/2626229689 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdbbc0862d0 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.505 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.503+0000 7fdbc67fc700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdbb0068f60 0x7fdbb006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:45.506 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.503+0000 7fdbc67fc700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdbb0068f60 0x7fdbb006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800b3b0 tx=0x7fdbb8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:45.507 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.507+0000 7fdbaf7fe700 1 -- 172.21.15.79:0/2626229689 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdbbc04ca00 con 0x7fdbc80fcbf0 2024-01-19T01:20:45.707 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.706+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbc810f520 con 0x7fdbb0068f60 2024-01-19T01:20:45.708 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.707+0000 7fdbaf7fe700 1 -- 172.21.15.79:0/2626229689 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fdbc810f520 con 0x7fdbb0068f60 2024-01-19T01:20:45.708 INFO:teuthology.orchestra.run.smithi079.stdout:Backend: cephadm 2024-01-19T01:20:45.708 INFO:teuthology.orchestra.run.smithi079.stdout:Available: Yes 2024-01-19T01:20:45.708 INFO:teuthology.orchestra.run.smithi079.stdout:Paused: No 2024-01-19T01:20:45.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdbb0068f60 msgr2=0x7fdbb006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:45.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdbb0068f60 0x7fdbb006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800b3b0 tx=0x7fdbb8009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:45.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fcbf0 msgr2=0x7fdbc810d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:45.710 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fcbf0 0x7fdbc810d4d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc002a80 tx=0x7fdbbc0046b0 comp rx=0 tx=0).stop 2024-01-19T01:20:45.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 shutdown_connections 2024-01-19T01:20:45.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdbb0068f60 0x7fdbb006b410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:45.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdbc80fcbf0 0x7fdbc810d4d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:45.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 --2- 172.21.15.79:0/2626229689 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc80fde00 0x7fdbc810da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:45.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 >> 172.21.15.79:0/2626229689 conn(0x7fdbc80f82d0 msgr2=0x7fdbc8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:45.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 shutdown_connections 2024-01-19T01:20:45.711 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:45.709+0000 7fdbcd47d700 1 -- 172.21.15.79:0/2626229689 wait complete. 2024-01-19T01:20:46.255 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph orch ps' 2024-01-19T01:20:46.508 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:46.807 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:46 smithi079 ceph-mon[28374]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:46.807 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:46 smithi079 ceph-mon[28374]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:46.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:46 smithi152 ceph-mon[35694]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:46.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:46 smithi152 ceph-mon[35694]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:48.234 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.231+0000 7fe12dde4700 1 -- 172.21.15.79:0/2906568014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280fdf60 msgr2=0x7fe1280fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.231+0000 7fe12dde4700 1 --2- 172.21.15.79:0/2906568014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280fdf60 0x7fe1280fe3b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe118004940 tx=0x7fe118009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 -- 172.21.15.79:0/2906568014 shutdown_connections 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 --2- 172.21.15.79:0/2906568014 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280fdf60 0x7fe1280fe3b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 --2- 172.21.15.79:0/2906568014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1280faed0 0x7fe1280fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 -- 172.21.15.79:0/2906568014 >> 172.21.15.79:0/2906568014 conn(0x7fe1280f6950 msgr2=0x7fe1280f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 -- 172.21.15.79:0/2906568014 shutdown_connections 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 -- 172.21.15.79:0/2906568014 wait complete. 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 Processor -- start 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.232+0000 7fe12dde4700 1 -- start start 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe12dde4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280faed0 0x7fe12810d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe12dde4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1280fdf60 0x7fe12810db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:48.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe12dde4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810e190 con 0x7fe1280faed0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe12dde4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12810ea30 con 0x7fe1280fdf60 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe1277fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280faed0 0x7fe12810d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe1277fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280faed0 0x7fe12810d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:51632/0 (socket says 172.21.15.79:51632) 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe1277fe700 1 -- 172.21.15.79:0/834489189 learned_addr learned my addr 172.21.15.79:0/834489189 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe126ffd700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1280fdf60 0x7fe12810db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe1277fe700 1 -- 172.21.15.79:0/834489189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1280fdf60 msgr2=0x7fe12810db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe1277fe700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1280fdf60 0x7fe12810db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe1277fe700 1 -- 172.21.15.79:0/834489189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118005040 con 0x7fe1280faed0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe1277fe700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280faed0 0x7fe12810d650 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe11c002a80 tx=0x7fe11c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe10ffff700 1 -- 172.21.15.79:0/834489189 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11c007cc0 con 0x7fe1280faed0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12810ed10 con 0x7fe1280faed0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe10ffff700 1 -- 172.21.15.79:0/834489189 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe11c00f040 con 0x7fe1280faed0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe10ffff700 1 -- 172.21.15.79:0/834489189 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11c00da20 con 0x7fe1280faed0 2024-01-19T01:20:48.236 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.233+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12810f250 con 0x7fe1280faed0 2024-01-19T01:20:48.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.234+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1281075f0 con 0x7fe1280faed0 2024-01-19T01:20:48.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.235+0000 7fe10ffff700 1 -- 172.21.15.79:0/834489189 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe11c00db80 con 0x7fe1280faed0 2024-01-19T01:20:48.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.235+0000 7fe10ffff700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe110068f60 0x7fe11006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:48.237 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.235+0000 7fe10ffff700 1 -- 172.21.15.79:0/834489189 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe11c086840 con 0x7fe1280faed0 2024-01-19T01:20:48.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.235+0000 7fe126ffd700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe110068f60 0x7fe11006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:48.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.236+0000 7fe126ffd700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe110068f60 0x7fe11006b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe118004f90 tx=0x7fe118009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:48.240 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.239+0000 7fe10ffff700 1 -- 172.21.15.79:0/834489189 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe11c056a60 con 0x7fe1280faed0 2024-01-19T01:20:48.433 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.433+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe12810f470 con 0x7fe110068f60 2024-01-19T01:20:48.442 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.441+0000 7fe10ffff700 1 -- 172.21.15.79:0/834489189 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7fe12810f470 con 0x7fe110068f60 2024-01-19T01:20:48.442 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.smithi079 smithi079 *:9093,9094 running (3m) 2m ago 5m 26.8M - 0.25.0 c8568f914cd2 e31a5a73bdf6 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi079 smithi079 running (5m) 2m ago 5m 6665k - 17.2.7-268-g6441305d c3a232f785d8 3b02e2e6b9be 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi152 smithi152 running (4m) 66s ago 4m 6660k - 17.2.7-268-g6441305d c3a232f785d8 b47c25b14963 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.smithi079 smithi079 *:3000 running (3m) 2m ago 4m 79.1M - 9.4.7 954c08fa6188 551c78ecdb2b 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi079.miqczk smithi079 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-268-g6441305d c3a232f785d8 d038b98fa75a 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi152.flyowb smithi152 *:8443,9283 running (4m) 66s ago 4m 432M - 17.2.7-268-g6441305d c3a232f785d8 92e644b493a7 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi079 smithi079 running (6m) 2m ago 6m 45.0M 2048M 17.2.7-268-g6441305d c3a232f785d8 0aba73c41ca9 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi152 smithi152 running (4m) 66s ago 4m 40.8M 2048M 17.2.7-268-g6441305d c3a232f785d8 33474ce60d80 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi079 smithi079 *:9100 running (5m) 2m ago 5m 15.7M - 1.5.0 0da6a335fe13 13236a399b50 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi152 smithi152 *:9100 running (4m) 66s ago 4m 16.2M - 1.5.0 0da6a335fe13 bc329f5a11bd 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (3m) 2m ago 3m 38.8M 4288M 17.2.7-268-g6441305d c3a232f785d8 07dc267fdcd6 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (2m) 2m ago 2m 39.0M 4288M 17.2.7-268-g6441305d c3a232f785d8 76e91cff255f 2024-01-19T01:20:48.443 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (2m) 2m ago 2m 38.4M 4288M 17.2.7-268-g6441305d c3a232f785d8 1fd04b4d8cd2 2024-01-19T01:20:48.444 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi152 running (2m) 66s ago 2m 38.3M 5312M 17.2.7-268-g6441305d c3a232f785d8 51d81c3e87b4 2024-01-19T01:20:48.444 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi152 running (96s) 66s ago 96s 41.7M 5312M 17.2.7-268-g6441305d c3a232f785d8 df8e00ca007c 2024-01-19T01:20:48.444 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi152 running (71s) 66s ago 71s 10.9M 5312M 17.2.7-268-g6441305d c3a232f785d8 81f6d0444b64 2024-01-19T01:20:48.444 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.smithi079 smithi079 *:9095 running (3m) 2m ago 4m 37.9M - 2.43.0 a07b618ecd1d bbe57c5d745e 2024-01-19T01:20:48.444 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe110068f60 msgr2=0x7fe11006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe110068f60 0x7fe11006b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe118004f90 tx=0x7fe118009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280faed0 msgr2=0x7fe12810d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280faed0 0x7fe12810d650 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe11c002a80 tx=0x7fe11c004600 comp rx=0 tx=0).stop 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 shutdown_connections 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe110068f60 0x7fe11006b410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe1280faed0 0x7fe12810d650 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 --2- 172.21.15.79:0/834489189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1280fdf60 0x7fe12810db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.443+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 >> 172.21.15.79:0/834489189 conn(0x7fe1280f6950 msgr2=0x7fe1280f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:48.445 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.444+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 shutdown_connections 2024-01-19T01:20:48.446 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:48.444+0000 7fe12dde4700 1 -- 172.21.15.79:0/834489189 wait complete. 2024-01-19T01:20:48.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:48 smithi079 ceph-mon[28374]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:48.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:48 smithi152 ceph-mon[35694]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:49.104 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph orch ls' 2024-01-19T01:20:49.362 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:49.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:49 smithi079 ceph-mon[28374]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:49.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:49.920 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:49 smithi152 ceph-mon[35694]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:49.920 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:20:50.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:50 smithi079 ceph-mon[28374]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:50.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:50 smithi152 ceph-mon[35694]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.012+0000 7f1038456700 1 -- 172.21.15.79:0/542601655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 msgr2=0x7f10300fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.012+0000 7f1038456700 1 --2- 172.21.15.79:0/542601655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f10300fbf00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f102c004940 tx=0x7f102c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 -- 172.21.15.79:0/542601655 shutdown_connections 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 --2- 172.21.15.79:0/542601655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10300fcd00 0x7f10300fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 --2- 172.21.15.79:0/542601655 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f10300fbf00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 -- 172.21.15.79:0/542601655 >> 172.21.15.79:0/542601655 conn(0x7f10300f71d0 msgr2=0x7f10300f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 -- 172.21.15.79:0/542601655 shutdown_connections 2024-01-19T01:20:51.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 -- 172.21.15.79:0/542601655 wait complete. 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 Processor -- start 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 -- start start 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f103010d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10300fcd00 0x7f103010dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010e140 con 0x7f10300fbaf0 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f1038456700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010e980 con 0x7f10300fcd00 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f10361f2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f103010d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f10361f2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f103010d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49592/0 (socket says 172.21.15.79:49592) 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.013+0000 7f10361f2700 1 -- 172.21.15.79:0/1752779012 learned_addr learned my addr 172.21.15.79:0/1752779012 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f10359f1700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10300fcd00 0x7f103010dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f10361f2700 1 -- 172.21.15.79:0/1752779012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10300fcd00 msgr2=0x7f103010dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f10361f2700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10300fcd00 0x7f103010dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f10361f2700 1 -- 172.21.15.79:0/1752779012 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c005040 con 0x7f10300fbaf0 2024-01-19T01:20:51.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f10361f2700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f103010d570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f102c004f90 tx=0x7f102c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:51.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f1026ffd700 1 -- 172.21.15.79:0/1752779012 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102c00e040 con 0x7f10300fbaf0 2024-01-19T01:20:51.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f1026ffd700 1 -- 172.21.15.79:0/1752779012 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f102c00bcd0 con 0x7f10300fbaf0 2024-01-19T01:20:51.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f1026ffd700 1 -- 172.21.15.79:0/1752779012 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102c01c630 con 0x7f10300fbaf0 2024-01-19T01:20:51.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103010ec00 con 0x7f10300fbaf0 2024-01-19T01:20:51.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.014+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103010f140 con 0x7f10300fbaf0 2024-01-19T01:20:51.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.015+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103004bb80 con 0x7f10300fbaf0 2024-01-19T01:20:51.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.015+0000 7f1026ffd700 1 -- 172.21.15.79:0/1752779012 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f102c013960 con 0x7f10300fbaf0 2024-01-19T01:20:51.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.016+0000 7f1026ffd700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f101c068f60 0x7f101c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:51.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.016+0000 7f10359f1700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f101c068f60 0x7f101c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:51.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.016+0000 7f1026ffd700 1 -- 172.21.15.79:0/1752779012 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f102c087ef0 con 0x7f10300fbaf0 2024-01-19T01:20:51.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.016+0000 7f10359f1700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f101c068f60 0x7f101c06b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1020006fd0 tx=0x7f1020008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:51.021 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.020+0000 7f1026ffd700 1 -- 172.21.15.79:0/1752779012 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f102c058190 con 0x7f10300fbaf0 2024-01-19T01:20:51.212 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.210+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f103010f560 con 0x7f101c068f60 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.214+0000 7f1026ffd700 1 -- 172.21.15.79:0/1752779012 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f103010f560 con 0x7f101c068f60 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:crash 2/2 2m ago 5m * 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:mgr 2/2 2m ago 6m count:2 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:mon 2/2 2m ago 5m smithi079:172.21.15.79=smithi079;smithi152:172.21.15.152=smithi152;count:2 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-01-19T01:20:51.216 INFO:teuthology.orchestra.run.smithi079.stdout:osd 6 2m ago - 2024-01-19T01:20:51.217 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f101c068f60 msgr2=0x7f101c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f101c068f60 0x7f101c06b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1020006fd0 tx=0x7f1020008040 comp rx=0 tx=0).stop 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 msgr2=0x7f103010d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f103010d570 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f102c004f90 tx=0x7f102c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 shutdown_connections 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f101c068f60 0x7f101c06b410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f10300fbaf0 0x7f103010d570 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:51.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 --2- 172.21.15.79:0/1752779012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f10300fcd00 0x7f103010dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:51.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 >> 172.21.15.79:0/1752779012 conn(0x7f10300f71d0 msgr2=0x7f1030065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:51.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 shutdown_connections 2024-01-19T01:20:51.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:51.217+0000 7f1038456700 1 -- 172.21.15.79:0/1752779012 wait complete. 2024-01-19T01:20:51.958 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph orch host ls' 2024-01-19T01:20:52.209 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:52.628 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:52 smithi079 ceph-mon[28374]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:52.629 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:52 smithi079 ceph-mon[28374]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:52.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:52 smithi152 ceph-mon[35694]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:52.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:52 smithi152 ceph-mon[35694]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 -- 172.21.15.79:0/1909064012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fec50 msgr2=0x7f31740fad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 --2- 172.21.15.79:0/1909064012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fec50 0x7f31740fad00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f315c004940 tx=0x7f315c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 -- 172.21.15.79:0/1909064012 shutdown_connections 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 --2- 172.21.15.79:0/1909064012 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fec50 0x7f31740fad00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 --2- 172.21.15.79:0/1909064012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31740fe270 0x7f31740fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 -- 172.21.15.79:0/1909064012 >> 172.21.15.79:0/1909064012 conn(0x7f31740f6950 msgr2=0x7f31740f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 -- 172.21.15.79:0/1909064012 shutdown_connections 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 -- 172.21.15.79:0/1909064012 wait complete. 2024-01-19T01:20:53.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.903+0000 7f317a061700 1 Processor -- start 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f317a061700 1 -- start start 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f317a061700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fe270 0x7f317410d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f317a061700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f317410db70 0x7f317410ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f317a061700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317410e0c0 con 0x7f31740fe270 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f317a061700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317410e200 con 0x7f317410db70 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f31737fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fe270 0x7f317410d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f31737fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fe270 0x7f317410d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49612/0 (socket says 172.21.15.79:49612) 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f31737fe700 1 -- 172.21.15.79:0/3315978790 learned_addr learned my addr 172.21.15.79:0/3315978790 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f316affd700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f317410db70 0x7f317410ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f31737fe700 1 -- 172.21.15.79:0/3315978790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f317410db70 msgr2=0x7f317410ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f31737fe700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f317410db70 0x7f317410ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f31737fe700 1 -- 172.21.15.79:0/3315978790 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c005040 con 0x7f31740fe270 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f316affd700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f317410db70 0x7f317410ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.904+0000 7f31737fe700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fe270 0x7f317410d630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f316400b790 tx=0x7f3164009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:53.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.905+0000 7f3170ff9700 1 -- 172.21.15.79:0/3315978790 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f316400f6f0 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.905+0000 7f3170ff9700 1 -- 172.21.15.79:0/3315978790 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f316400fd00 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.905+0000 7f3170ff9700 1 -- 172.21.15.79:0/3315978790 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31640103a0 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.905+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317410f0f0 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.905+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317410f5e0 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.906+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31741076a0 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.906+0000 7f3170ff9700 1 -- 172.21.15.79:0/3315978790 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f316400f850 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.906+0000 7f3170ff9700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3160068f60 0x7f316006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.906+0000 7f3170ff9700 1 -- 172.21.15.79:0/3315978790 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3164086890 con 0x7f31740fe270 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.906+0000 7f316affd700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3160068f60 0x7f316006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:53.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.907+0000 7f316affd700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3160068f60 0x7f316006b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:53.913 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:53.910+0000 7f3170ff9700 1 -- 172.21.15.79:0/3315978790 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3164056ab0 con 0x7f31740fe270 2024-01-19T01:20:54.099 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.098+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f31740fec50 con 0x7f3160068f60 2024-01-19T01:20:54.101 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.100+0000 7f3170ff9700 1 -- 172.21.15.79:0/3315978790 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f31740fec50 con 0x7f3160068f60 2024-01-19T01:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout:HOST ADDR LABELS STATUS 2024-01-19T01:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 172.21.15.79 2024-01-19T01:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout:smithi152 172.21.15.152 2024-01-19T01:20:54.101 INFO:teuthology.orchestra.run.smithi079.stdout:2 hosts in cluster 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3160068f60 msgr2=0x7f316006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3160068f60 0x7f316006b410 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f315c000c00 tx=0x7f315c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fe270 msgr2=0x7f317410d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fe270 0x7f317410d630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f316400b790 tx=0x7f3164009220 comp rx=0 tx=0).stop 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 shutdown_connections 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3160068f60 0x7f316006b410 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31740fe270 0x7f317410d630 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 --2- 172.21.15.79:0/3315978790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f317410db70 0x7f317410ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 >> 172.21.15.79:0/3315978790 conn(0x7f31740f6950 msgr2=0x7f31740f8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:54.103 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 shutdown_connections 2024-01-19T01:20:54.104 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:54.102+0000 7f317a061700 1 -- 172.21.15.79:0/3315978790 wait complete. 2024-01-19T01:20:54.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:54 smithi079 ceph-mon[28374]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:54.894 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph orch device ls' 2024-01-19T01:20:54.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:54 smithi152 ceph-mon[35694]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:55.151 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:55.876 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:55 smithi079 ceph-mon[28374]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:55.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:55 smithi152 ceph-mon[35694]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:56.868 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:56 smithi079 ceph-mon[28374]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:56.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.891+0000 7f381c53b700 1 -- 172.21.15.79:0/1299287651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 msgr2=0x7f38140fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:56.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.891+0000 7f381c53b700 1 --2- 172.21.15.79:0/1299287651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f38140fbf00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3810004940 tx=0x7f3810009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:56.894 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 -- 172.21.15.79:0/1299287651 shutdown_connections 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 --2- 172.21.15.79:0/1299287651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38140fcd00 0x7f38140fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 --2- 172.21.15.79:0/1299287651 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f38140fbf00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 -- 172.21.15.79:0/1299287651 >> 172.21.15.79:0/1299287651 conn(0x7f38140f71d0 msgr2=0x7f38140f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 -- 172.21.15.79:0/1299287651 shutdown_connections 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 -- 172.21.15.79:0/1299287651 wait complete. 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 Processor -- start 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 -- start start 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f3814110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38140fcd00 0x7f38141127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38140669a0 con 0x7f38140fbaf0 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381c53b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3814066ae0 con 0x7f38140fcd00 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381a2d7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f3814110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381a2d7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f3814110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:49628/0 (socket says 172.21.15.79:49628) 2024-01-19T01:20:56.895 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.892+0000 7f381a2d7700 1 -- 172.21.15.79:0/1244270024 learned_addr learned my addr 172.21.15.79:0/1244270024 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f3819ad6700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38140fcd00 0x7f38141127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f381a2d7700 1 -- 172.21.15.79:0/1244270024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38140fcd00 msgr2=0x7f38141127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f381a2d7700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38140fcd00 0x7f38141127e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f381a2d7700 1 -- 172.21.15.79:0/1244270024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3810005040 con 0x7f38140fbaf0 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f381a2d7700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f3814110290 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f3810009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f380affd700 1 -- 172.21.15.79:0/1244270024 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381000e040 con 0x7f38140fbaf0 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f380affd700 1 -- 172.21.15.79:0/1244270024 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3810009af0 con 0x7f38140fbaf0 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3814112de0 con 0x7f38140fbaf0 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f380affd700 1 -- 172.21.15.79:0/1244270024 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f381001c630 con 0x7f38140fbaf0 2024-01-19T01:20:56.896 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.893+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3814113240 con 0x7f38140fbaf0 2024-01-19T01:20:56.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.894+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381404bb80 con 0x7f38140fbaf0 2024-01-19T01:20:56.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.894+0000 7f380affd700 1 -- 172.21.15.79:0/1244270024 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f381000bc50 con 0x7f38140fbaf0 2024-01-19T01:20:56.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.895+0000 7f380affd700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3800068f60 0x7f380006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:56.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.895+0000 7f380affd700 1 -- 172.21.15.79:0/1244270024 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3810087db0 con 0x7f38140fbaf0 2024-01-19T01:20:56.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.895+0000 7f3819ad6700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3800068f60 0x7f380006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:56.897 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.895+0000 7f3819ad6700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3800068f60 0x7f380006b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3804007fd0 tx=0x7f3804009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:56.900 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:56.899+0000 7f380affd700 1 -- 172.21.15.79:0/1244270024 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f38100545c0 con 0x7f38140fbaf0 2024-01-19T01:20:56.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:56 smithi152 ceph-mon[35694]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:57.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.090+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3814101650 con 0x7f3800068f60 2024-01-19T01:20:57.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.091+0000 7f380affd700 1 -- 172.21.15.79:0/1244270024 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f3814101650 con 0x7f3800068f60 2024-01-19T01:20:57.092 INFO:teuthology.orchestra.run.smithi079.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-01-19T01:20:57.092 INFO:teuthology.orchestra.run.smithi079.stdout:smithi079 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013G400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-01-19T01:20:57.092 INFO:teuthology.orchestra.run.smithi079.stdout:smithi152 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JT400BGN 372G No 71s ago Has a FileSystem, LVM detected 2024-01-19T01:20:57.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3800068f60 msgr2=0x7f380006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:57.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3800068f60 0x7f380006b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3804007fd0 tx=0x7f3804009040 comp rx=0 tx=0).stop 2024-01-19T01:20:57.094 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 msgr2=0x7f3814110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f3814110290 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f3810009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 shutdown_connections 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3800068f60 0x7f380006b410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f38140fbaf0 0x7f3814110290 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 --2- 172.21.15.79:0/1244270024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38140fcd00 0x7f38141127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.093+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 >> 172.21.15.79:0/1244270024 conn(0x7f38140f71d0 msgr2=0x7f38140fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.094+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 shutdown_connections 2024-01-19T01:20:57.095 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:57.094+0000 7f381c53b700 1 -- 172.21.15.79:0/1244270024 wait complete. 2024-01-19T01:20:57.756 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T01:20:57.765 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:20:57.766 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-01-19T01:20:58.019 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:20:58.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:58 smithi079 ceph-mon[28374]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:58.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:20:58 smithi079 ceph-mon[28374]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:58.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:58 smithi152 ceph-mon[35694]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:20:58.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:20:58 smithi152 ceph-mon[35694]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:20:59.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 -- 172.21.15.79:0/359773138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fcd90 msgr2=0x7fb9700fd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:59.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 --2- 172.21.15.79:0/359773138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fcd90 0x7fb9700fd200 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb95c004940 tx=0x7fb95c009e30 comp rx=0 tx=0).stop 2024-01-19T01:20:59.778 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 -- 172.21.15.79:0/359773138 shutdown_connections 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 --2- 172.21.15.79:0/359773138 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fcd90 0x7fb9700fd200 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 --2- 172.21.15.79:0/359773138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9700fbae0 0x7fb9700fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 -- 172.21.15.79:0/359773138 >> 172.21.15.79:0/359773138 conn(0x7fb9700f7240 msgr2=0x7fb9700f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 -- 172.21.15.79:0/359773138 shutdown_connections 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 -- 172.21.15.79:0/359773138 wait complete. 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 Processor -- start 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.776+0000 7fb976fe0700 1 -- start start 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb976fe0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fbae0 0x7fb97010f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb976fe0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9700fcd90 0x7fb97010fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb976fe0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970110300 con 0x7fb9700fbae0 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb976fe0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970110b40 con 0x7fb9700fcd90 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb974d7c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fbae0 0x7fb97010f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb974d7c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fbae0 0x7fb97010f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45690/0 (socket says 172.21.15.79:45690) 2024-01-19T01:20:59.779 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb974d7c700 1 -- 172.21.15.79:0/691438671 learned_addr learned my addr 172.21.15.79:0/691438671 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb96ffff700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9700fcd90 0x7fb97010fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb974d7c700 1 -- 172.21.15.79:0/691438671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9700fcd90 msgr2=0x7fb97010fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb974d7c700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9700fcd90 0x7fb97010fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb974d7c700 1 -- 172.21.15.79:0/691438671 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c005040 con 0x7fb9700fbae0 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb974d7c700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fbae0 0x7fb97010f730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb96400b790 tx=0x7fb964009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb96d7fa700 1 -- 172.21.15.79:0/691438671 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb96400f6f0 con 0x7fb9700fbae0 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb970110e20 con 0x7fb9700fbae0 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb96d7fa700 1 -- 172.21.15.79:0/691438671 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb96400fd00 con 0x7fb9700fbae0 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb96d7fa700 1 -- 172.21.15.79:0/691438671 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9640103a0 con 0x7fb9700fbae0 2024-01-19T01:20:59.780 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.777+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb970111360 con 0x7fb9700fbae0 2024-01-19T01:20:59.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.778+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97004bb80 con 0x7fb9700fbae0 2024-01-19T01:20:59.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.779+0000 7fb96d7fa700 1 -- 172.21.15.79:0/691438671 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb96400f850 con 0x7fb9700fbae0 2024-01-19T01:20:59.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.779+0000 7fb96d7fa700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb960068f60 0x7fb96006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:20:59.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.779+0000 7fb96d7fa700 1 -- 172.21.15.79:0/691438671 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb964086840 con 0x7fb9700fbae0 2024-01-19T01:20:59.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.779+0000 7fb96ffff700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb960068f60 0x7fb96006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:20:59.781 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.780+0000 7fb96ffff700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb960068f60 0x7fb96006b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb95c00b3b0 tx=0x7fb95c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:20:59.784 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.784+0000 7fb96d7fa700 1 -- 172.21.15.79:0/691438671 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb964056ae0 con 0x7fb9700fbae0 2024-01-19T01:20:59.996 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:20:59.995+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb970066330 con 0x7fb960068f60 2024-01-19T01:21:01.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:00 smithi152 ceph-mon[35694]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:01.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:00 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-01-19T01:21:01.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:00 smithi079 ceph-mon[28374]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:01.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:00 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-01-19T01:21:01.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.606+0000 7fb96d7fa700 1 -- 172.21.15.79:0/691438671 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb970066330 con 0x7fb960068f60 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb960068f60 msgr2=0x7fb96006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb960068f60 0x7fb96006b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb95c00b3b0 tx=0x7fb95c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fbae0 msgr2=0x7fb97010f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fbae0 0x7fb97010f730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb96400b790 tx=0x7fb964009250 comp rx=0 tx=0).stop 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 shutdown_connections 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb960068f60 0x7fb96006b410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb9700fbae0 0x7fb97010f730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 --2- 172.21.15.79:0/691438671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9700fcd90 0x7fb97010fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 >> 172.21.15.79:0/691438671 conn(0x7fb9700f7240 msgr2=0x7fb970065c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 shutdown_connections 2024-01-19T01:21:01.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:01.609+0000 7fb976fe0700 1 -- 172.21.15.79:0/691438671 wait complete. 2024-01-19T01:21:01.960 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-f3747e56-b667-11ee-95ae-87774f69a715-mon-smithi079[28370]: 2024-01-19T01:21:01.584+0000 7ff685e9c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: osdmap e41: 6 total, 6 up, 6 in 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: pgmap v170: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: osdmap e42: 6 total, 6 up, 6 in 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-19T01:21:01.963 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: osdmap e43: 6 total, 6 up, 6 in 2024-01-19T01:21:01.964 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-01-19T01:21:01.964 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: fsmap cephfs:0 2024-01-19T01:21:01.964 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: Saving service mds.cephfs spec with placement count:4 2024-01-19T01:21:01.964 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:01.964 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:01 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:02.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:21:02.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-01-19T01:21:02.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: osdmap e41: 6 total, 6 up, 6 in 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: pgmap v170: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: osdmap e42: 6 total, 6 up, 6 in 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: osdmap e43: 6 total, 6 up, 6 in 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: fsmap cephfs:0 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: Saving service mds.cephfs spec with placement count:4 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:02.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:01 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:02.304 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph fs dump' 2024-01-19T01:21:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-19T01:21:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:02.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.cwngbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:02.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.cwngbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:02.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:02.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: Deploying daemon mds.cephfs.smithi079.cwngbt on smithi079 2024-01-19T01:21:02.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:02 smithi079 ceph-mon[28374]: osdmap e44: 6 total, 6 up, 6 in 2024-01-19T01:21:03.035 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:03.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.cwngbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.cwngbt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: Deploying daemon mds.cephfs.smithi079.cwngbt on smithi079 2024-01-19T01:21:03.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:02 smithi152 ceph-mon[35694]: osdmap e44: 6 total, 6 up, 6 in 2024-01-19T01:21:04.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:04 smithi079 ceph-mon[28374]: pgmap v174: 65 pgs: 52 unknown, 13 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:04.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:04 smithi079 ceph-mon[28374]: osdmap e45: 6 total, 6 up, 6 in 2024-01-19T01:21:04.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:04 smithi079 ceph-mon[28374]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-19T01:21:04.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:04.692 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:04 smithi152 ceph-mon[35694]: pgmap v174: 65 pgs: 52 unknown, 13 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:04 smithi152 ceph-mon[35694]: osdmap e45: 6 total, 6 up, 6 in 2024-01-19T01:21:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:04 smithi152 ceph-mon[35694]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-19T01:21:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:04.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 -- 172.21.15.79:0/1976170681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 msgr2=0x7feac00fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 --2- 172.21.15.79:0/1976170681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac00fe250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feab0004940 tx=0x7feab0009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 -- 172.21.15.79:0/1976170681 shutdown_connections 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 --2- 172.21.15.79:0/1976170681 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac00fe250 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 --2- 172.21.15.79:0/1976170681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac00fcbf0 0x7feac00fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 -- 172.21.15.79:0/1976170681 >> 172.21.15.79:0/1976170681 conn(0x7feac00f82d0 msgr2=0x7feac00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 -- 172.21.15.79:0/1976170681 shutdown_connections 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.447+0000 7feac55b4700 1 -- 172.21.15.79:0/1976170681 wait complete. 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feac55b4700 1 Processor -- start 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feac55b4700 1 -- start start 2024-01-19T01:21:05.560 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feac55b4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac00fcbf0 0x7feac0071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feac55b4700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feac55b4700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0070620 con 0x7feac00fde00 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feac55b4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0070790 con 0x7feac00fcbf0 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabe7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabe7fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac006ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45718/0 (socket says 172.21.15.79:45718) 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabe7fc700 1 -- 172.21.15.79:0/1852349096 learned_addr learned my addr 172.21.15.79:0/1852349096 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabeffd700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac00fcbf0 0x7feac0071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabe7fc700 1 -- 172.21.15.79:0/1852349096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac00fcbf0 msgr2=0x7feac0071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabe7fc700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac00fcbf0 0x7feac0071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabe7fc700 1 -- 172.21.15.79:0/1852349096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0005040 con 0x7feac00fde00 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feabe7fc700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac006ff90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feab0009380 tx=0x7feab0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feaa77fe700 1 -- 172.21.15.79:0/1852349096 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab000e040 con 0x7feac00fde00 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.448+0000 7feaa77fe700 1 -- 172.21.15.79:0/1852349096 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feab000bd20 con 0x7feac00fde00 2024-01-19T01:21:05.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.449+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac0070a10 con 0x7feac00fde00 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.449+0000 7feaa77fe700 1 -- 172.21.15.79:0/1852349096 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab001c630 con 0x7feac00fde00 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.449+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac01155a0 con 0x7feac00fde00 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.449+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac004bb80 con 0x7feac00fde00 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.450+0000 7feaa77fe700 1 -- 172.21.15.79:0/1852349096 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7feab0013420 con 0x7feac00fde00 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.450+0000 7feaa77fe700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7feaa8068f60 0x7feaa806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.450+0000 7feaa77fe700 1 -- 172.21.15.79:0/1852349096 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7feab0088130 con 0x7feac00fde00 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.450+0000 7feabeffd700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7feaa8068f60 0x7feaa806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.451+0000 7feabeffd700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7feaa8068f60 0x7feaa806b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feac00718a0 tx=0x7feab40091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:05.562 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.455+0000 7feaa77fe700 1 -- 172.21.15.79:0/1852349096 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7feab0058090 con 0x7feac00fde00 2024-01-19T01:21:05.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.696+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7feac0115920 con 0x7feac00fde00 2024-01-19T01:21:05.699 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.699+0000 7feaa77fe700 1 -- 172.21.15.79:0/1852349096 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7feab0057c20 con 0x7feac00fde00 2024-01-19T01:21:05.699 INFO:teuthology.orchestra.run.smithi079.stdout:e4 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 4 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-01-19T01:21:01.585027+0000 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-01-19T01:21:05.601521+0000 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 60 2024-01-19T01:21:05.700 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 0 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 1 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:in 0 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:up {0=14460} 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data disabled 2024-01-19T01:21:05.701 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-01-19T01:21:05.702 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 0 2024-01-19T01:21:05.702 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.cwngbt{0:14460} state up:creating seq 1 addr [v2:172.21.15.79:6826/1029843807,v1:172.21.15.79:6827/1029843807] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:21:05.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:05.702 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:05.702 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.700+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7feaa8068f60 msgr2=0x7feaa806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.700+0000 7feac55b4700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7feaa8068f60 0x7feaa806b410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7feac00718a0 tx=0x7feab40091a0 comp rx=0 tx=0).stop 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 msgr2=0x7feac006ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac006ff90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feab0009380 tx=0x7feab0009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 shutdown_connections 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7feaa8068f60 0x7feaa806b410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feac00fcbf0 0x7feac0071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 --2- 172.21.15.79:0/1852349096 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7feac00fde00 0x7feac006ff90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 >> 172.21.15.79:0/1852349096 conn(0x7feac00f82d0 msgr2=0x7feac0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 shutdown_connections 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:05.701+0000 7feac55b4700 1 -- 172.21.15.79:0/1852349096 wait complete. 2024-01-19T01:21:05.703 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 4 2024-01-19T01:21:06.413 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T01:21:06.435 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:21:06.436 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-01-19T01:21:06.696 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:06.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: pgmap v176: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6826/1029843807,v1:172.21.15.79:6827/1029843807] up:boot 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: daemon mds.cephfs.smithi079.cwngbt assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: Cluster is now healthy 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: fsmap cephfs:0 1 up:standby 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.cwngbt"}]: dispatch 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:creating} 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: daemon mds.cephfs.smithi079.cwngbt is now active in filesystem cephfs as rank 0 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1852349096' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:21:06.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:06.750 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.hpxjws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:06.750 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.hpxjws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:06.750 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:06.750 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:06 smithi079 ceph-mon[28374]: Deploying daemon mds.cephfs.smithi152.hpxjws on smithi152 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: pgmap v176: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6826/1029843807,v1:172.21.15.79:6827/1029843807] up:boot 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: daemon mds.cephfs.smithi079.cwngbt assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: Cluster is now healthy 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: fsmap cephfs:0 1 up:standby 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.cwngbt"}]: dispatch 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:creating} 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: daemon mds.cephfs.smithi079.cwngbt is now active in filesystem cephfs as rank 0 2024-01-19T01:21:06.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:06.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:06.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1852349096' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:21:06.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:06.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.hpxjws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:06.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.hpxjws", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:06.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:06.925 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:06 smithi152 ceph-mon[35694]: Deploying daemon mds.cephfs.smithi152.hpxjws on smithi152 2024-01-19T01:21:07.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:07 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6826/1029843807,v1:172.21.15.79:6827/1029843807] up:active 2024-01-19T01:21:07.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:07 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 2024-01-19T01:21:07.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:07 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6826/1029843807,v1:172.21.15.79:6827/1029843807] up:active 2024-01-19T01:21:07.945 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:07 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 2024-01-19T01:21:08.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.241+0000 7f6367c72700 1 -- 172.21.15.79:0/998296837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fcbf0 msgr2=0x7f63600fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:08.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.241+0000 7f6367c72700 1 --2- 172.21.15.79:0/998296837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fcbf0 0x7f63600fd000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f635c004940 tx=0x7f635c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:08.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.241+0000 7f6367c72700 1 -- 172.21.15.79:0/998296837 shutdown_connections 2024-01-19T01:21:08.244 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.241+0000 7f6367c72700 1 --2- 172.21.15.79:0/998296837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63600fde00 0x7f63600fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.241+0000 7f6367c72700 1 --2- 172.21.15.79:0/998296837 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fcbf0 0x7f63600fd000 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.241+0000 7f6367c72700 1 -- 172.21.15.79:0/998296837 >> 172.21.15.79:0/998296837 conn(0x7f63600f82d0 msgr2=0x7f63600fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.241+0000 7f6367c72700 1 -- 172.21.15.79:0/998296837 shutdown_connections 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6367c72700 1 -- 172.21.15.79:0/998296837 wait complete. 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6367c72700 1 Processor -- start 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6367c72700 1 -- start start 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6367c72700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63600fcbf0 0x7f636010d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6367c72700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fde00 0x7f636010da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6367c72700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636010e010 con 0x7f63600fde00 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6367c72700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636010e8b0 con 0x7f63600fcbf0 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f636520d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fde00 0x7f636010da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f636520d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fde00 0x7f636010da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:45756/0 (socket says 172.21.15.79:45756) 2024-01-19T01:21:08.245 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f636520d700 1 -- 172.21.15.79:0/1361581163 learned_addr learned my addr 172.21.15.79:0/1361581163 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.242+0000 7f6365a0e700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63600fcbf0 0x7f636010d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f636520d700 1 -- 172.21.15.79:0/1361581163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63600fcbf0 msgr2=0x7f636010d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f636520d700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63600fcbf0 0x7f636010d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f636520d700 1 -- 172.21.15.79:0/1361581163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c005040 con 0x7f63600fde00 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f636520d700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fde00 0x7f636010da10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6350002a80 tx=0x7f63500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f63567fc700 1 -- 172.21.15.79:0/1361581163 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6350007cc0 con 0x7f63600fde00 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f63567fc700 1 -- 172.21.15.79:0/1361581163 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f635000f040 con 0x7f63600fde00 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636010eb90 con 0x7f63600fde00 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f63567fc700 1 -- 172.21.15.79:0/1361581163 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f635000da60 con 0x7f63600fde00 2024-01-19T01:21:08.246 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.243+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636010f0d0 con 0x7f63600fde00 2024-01-19T01:21:08.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.244+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636004bb80 con 0x7f63600fde00 2024-01-19T01:21:08.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.244+0000 7f63567fc700 1 -- 172.21.15.79:0/1361581163 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f635000dc60 con 0x7f63600fde00 2024-01-19T01:21:08.247 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.245+0000 7f63567fc700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f634c068f60 0x7f634c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:08.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.245+0000 7f63567fc700 1 -- 172.21.15.79:0/1361581163 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6350086a20 con 0x7f63600fde00 2024-01-19T01:21:08.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.245+0000 7f6365a0e700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f634c068f60 0x7f634c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:08.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.245+0000 7f6365a0e700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f634c068f60 0x7f634c06b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f635c004890 tx=0x7f635c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:08.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.253+0000 7f63567fc700 1 -- 172.21.15.79:0/1361581163 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6350056900 con 0x7f63600fde00 2024-01-19T01:21:08.504 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.502+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f6360105590 con 0x7f63600fde00 2024-01-19T01:21:08.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.618+0000 7f63567fc700 1 -- 172.21.15.79:0/1361581163 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f6360105590 con 0x7f63600fde00 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f634c068f60 msgr2=0x7f634c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f634c068f60 0x7f634c06b410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f635c004890 tx=0x7f635c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fde00 msgr2=0x7f636010da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fde00 0x7f636010da10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6350002a80 tx=0x7f63500046b0 comp rx=0 tx=0).stop 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 shutdown_connections 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f634c068f60 0x7f634c06b410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f63600fcbf0 0x7f636010d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 --2- 172.21.15.79:0/1361581163 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f63600fde00 0x7f636010da10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:08.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 >> 172.21.15.79:0/1361581163 conn(0x7f63600f82d0 msgr2=0x7f6360101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:08.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 shutdown_connections 2024-01-19T01:21:08.623 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:08.621+0000 7f6367c72700 1 -- 172.21.15.79:0/1361581163 wait complete. 2024-01-19T01:21:08.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: pgmap v177: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:08.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:08.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:08.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:08.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.qxcrcz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:08.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.qxcrcz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:08.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:08.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:08 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1361581163' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-01-19T01:21:08.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: pgmap v177: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:08.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:08.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:08.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:08.949 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.qxcrcz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:08.950 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi079.qxcrcz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:08.950 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:08.950 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:08 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1361581163' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-01-19T01:21:09.880 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:09 smithi079 ceph-mon[28374]: Deploying daemon mds.cephfs.smithi079.qxcrcz on smithi079 2024-01-19T01:21:09.880 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:09 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] up:boot 2024-01-19T01:21:09.880 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:09 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1361581163' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-01-19T01:21:09.880 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:09 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 1 up:standby 2024-01-19T01:21:09.881 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:09 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.hpxjws"}]: dispatch 2024-01-19T01:21:09.881 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:09 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 1 up:standby 2024-01-19T01:21:09.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:09 smithi152 ceph-mon[35694]: Deploying daemon mds.cephfs.smithi079.qxcrcz on smithi079 2024-01-19T01:21:09.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:09 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] up:boot 2024-01-19T01:21:09.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:09 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1361581163' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-01-19T01:21:09.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:09 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 1 up:standby 2024-01-19T01:21:09.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:09 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.hpxjws"}]: dispatch 2024-01-19T01:21:09.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:09 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 1 up:standby 2024-01-19T01:21:10.575 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T01:21:10.584 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:21:10.585 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-01-19T01:21:10.846 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:10 smithi079 ceph-mon[28374]: pgmap v178: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 805 B/s wr, 2 op/s 2024-01-19T01:21:10.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:10 smithi152 ceph-mon[35694]: pgmap v178: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 805 B/s wr, 2 op/s 2024-01-19T01:21:11.266 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:12.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-01-19T01:21:12.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.btzecy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.btzecy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: Deploying daemon mds.cephfs.smithi152.btzecy on smithi152 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:boot 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 2 up:standby 2024-01-19T01:21:12.424 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.qxcrcz"}]: dispatch 2024-01-19T01:21:12.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.btzecy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi152.btzecy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: Deploying daemon mds.cephfs.smithi152.btzecy on smithi152 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:boot 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 2 up:standby 2024-01-19T01:21:12.499 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.qxcrcz"}]: dispatch 2024-01-19T01:21:12.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.858+0000 7f9fe32be700 1 -- 172.21.15.79:0/2939089844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0feb80 msgr2=0x7f9fdc0facc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:12.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.859+0000 7f9fe32be700 1 --2- 172.21.15.79:0/2939089844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0feb80 0x7f9fdc0facc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8004940 tx=0x7f9fd8009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:12.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.859+0000 7f9fe32be700 1 -- 172.21.15.79:0/2939089844 shutdown_connections 2024-01-19T01:21:12.861 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.859+0000 7f9fe32be700 1 --2- 172.21.15.79:0/2939089844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0feb80 0x7f9fdc0facc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.859+0000 7f9fe32be700 1 --2- 172.21.15.79:0/2939089844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fdc0fe230 0x7f9fdc0fe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.859+0000 7f9fe32be700 1 -- 172.21.15.79:0/2939089844 >> 172.21.15.79:0/2939089844 conn(0x7f9fdc0f69d0 msgr2=0x7f9fdc0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.859+0000 7f9fe32be700 1 -- 172.21.15.79:0/2939089844 shutdown_connections 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.859+0000 7f9fe32be700 1 -- 172.21.15.79:0/2939089844 wait complete. 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe32be700 1 Processor -- start 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe32be700 1 -- start start 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe32be700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0fe230 0x7f9fdc10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe32be700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fdc10df10 0x7f9fdc10edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe32be700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc10e3d0 con 0x7f9fdc0fe230 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe32be700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc10e510 con 0x7f9fdc10df10 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe105a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0fe230 0x7f9fdc10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe105a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0fe230 0x7f9fdc10d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57408/0 (socket says 172.21.15.79:57408) 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe105a700 1 -- 172.21.15.79:0/4251409304 learned_addr learned my addr 172.21.15.79:0/4251409304 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:12.862 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.860+0000 7f9fe0859700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fdc10df10 0x7f9fdc10edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:12.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fe105a700 1 -- 172.21.15.79:0/4251409304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fdc10df10 msgr2=0x7f9fdc10edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:12.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fe105a700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fdc10df10 0x7f9fdc10edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:12.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fe105a700 1 -- 172.21.15.79:0/4251409304 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd8005040 con 0x7f9fdc0fe230 2024-01-19T01:21:12.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fe105a700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0fe230 0x7f9fdc10d9d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00c8c0 tx=0x7f9fcc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:12.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fd1ffb700 1 -- 172.21.15.79:0/4251409304 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fcc00a760 con 0x7f9fdc0fe230 2024-01-19T01:21:12.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fd1ffb700 1 -- 172.21.15.79:0/4251409304 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fcc00ad70 con 0x7f9fdc0fe230 2024-01-19T01:21:12.864 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fdc10f410 con 0x7f9fdc0fe230 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fd1ffb700 1 -- 172.21.15.79:0/4251409304 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fcc0113a0 con 0x7f9fdc0fe230 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.861+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc10f920 con 0x7f9fdc0fe230 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.862+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc1075e0 con 0x7f9fdc0fe230 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.862+0000 7f9fd1ffb700 1 -- 172.21.15.79:0/4251409304 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9fcc011500 con 0x7f9fdc0fe230 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.863+0000 7f9fd1ffb700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9fc8068f60 0x7f9fc806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.863+0000 7f9fd1ffb700 1 -- 172.21.15.79:0/4251409304 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9fcc086e20 con 0x7f9fdc0fe230 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.863+0000 7f9fe0859700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9fc8068f60 0x7f9fc806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:12.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.863+0000 7f9fe0859700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9fc8068f60 0x7f9fc806b410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8004890 tx=0x7f9fd8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:12.868 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:12.867+0000 7f9fd1ffb700 1 -- 172.21.15.79:0/4251409304 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9fcc056df0 con 0x7f9fdc0fe230 2024-01-19T01:21:13.096 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.096+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9fdc10fba0 con 0x7f9fdc0fe230 2024-01-19T01:21:13.279 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.278+0000 7f9fd1ffb700 1 -- 172.21.15.79:0/4251409304 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9fcc056980 con 0x7f9fdc0fe230 2024-01-19T01:21:13.281 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.280+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9fc8068f60 msgr2=0x7f9fc806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.280+0000 7f9fe32be700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9fc8068f60 0x7f9fc806b410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8004890 tx=0x7f9fd8009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.280+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0fe230 msgr2=0x7f9fdc10d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.280+0000 7f9fe32be700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0fe230 0x7f9fdc10d9d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00c8c0 tx=0x7f9fcc0044c0 comp rx=0 tx=0).stop 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.281+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 shutdown_connections 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.281+0000 7f9fe32be700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f9fc8068f60 0x7f9fc806b410 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.281+0000 7f9fe32be700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f9fdc0fe230 0x7f9fdc10d9d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.281+0000 7f9fe32be700 1 --2- 172.21.15.79:0/4251409304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fdc10df10 0x7f9fdc10edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.281+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 >> 172.21.15.79:0/4251409304 conn(0x7f9fdc0f69d0 msgr2=0x7f9fdc101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.281+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 shutdown_connections 2024-01-19T01:21:13.282 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:13.281+0000 7f9fe32be700 1 -- 172.21.15.79:0/4251409304 wait complete. 2024-01-19T01:21:13.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:13 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/4251409304' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-01-19T01:21:13.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:13 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/4251409304' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-01-19T01:21:13.988 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T01:21:13.998 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:21:13.998 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-01-19T01:21:14.269 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:14.564 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-01-19T01:21:14.564 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/4251409304' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 2 up:standby 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:14.565 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:14 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-01-19T01:21:14.615 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/4251409304' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 2 up:standby 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:14.616 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:14 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:15.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:15 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] up:boot 2024-01-19T01:21:15.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:15 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 3 up:standby 2024-01-19T01:21:15.423 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:15 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.btzecy"}]: dispatch 2024-01-19T01:21:15.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:15 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] up:boot 2024-01-19T01:21:15.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:15 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 3 up:standby 2024-01-19T01:21:15.498 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:15 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi152.btzecy"}]: dispatch 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 -- 172.21.15.79:0/540954676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fbaf0 msgr2=0x7f17fc0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 --2- 172.21.15.79:0/540954676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fbaf0 0x7f17fc0fbf00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f17ec004940 tx=0x7f17ec009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 -- 172.21.15.79:0/540954676 shutdown_connections 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 --2- 172.21.15.79:0/540954676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17fc0fcd00 0x7f17fc0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 --2- 172.21.15.79:0/540954676 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fbaf0 0x7f17fc0fbf00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 -- 172.21.15.79:0/540954676 >> 172.21.15.79:0/540954676 conn(0x7f17fc0f7190 msgr2=0x7f17fc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 -- 172.21.15.79:0/540954676 shutdown_connections 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.450+0000 7f18044e7700 1 -- 172.21.15.79:0/540954676 wait complete. 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f18044e7700 1 Processor -- start 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f18044e7700 1 -- start start 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f18044e7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17fc0fbaf0 0x7f17fc112700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:16.453 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f18044e7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fcd00 0x7f17fc114c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f18044e7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc066000 con 0x7f17fc0fcd00 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f18044e7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc066140 con 0x7f17fc0fbaf0 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1801a82700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fcd00 0x7f17fc114c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1801a82700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fcd00 0x7f17fc114c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:57432/0 (socket says 172.21.15.79:57432) 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1801a82700 1 -- 172.21.15.79:0/3689930702 learned_addr learned my addr 172.21.15.79:0/3689930702 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1801a82700 1 -- 172.21.15.79:0/3689930702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17fc0fbaf0 msgr2=0x7f17fc112700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1802283700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17fc0fbaf0 0x7f17fc112700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1801a82700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17fc0fbaf0 0x7f17fc112700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1801a82700 1 -- 172.21.15.79:0/3689930702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec005040 con 0x7f17fc0fcd00 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1802283700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17fc0fbaf0 0x7f17fc112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f1801a82700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fcd00 0x7f17fc114c50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f17f800b790 tx=0x7f17f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f17f2ffd700 1 -- 172.21.15.79:0/3689930702 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f800f770 con 0x7f17fc0fcd00 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17fc115190 con 0x7f17fc0fcd00 2024-01-19T01:21:16.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f17f2ffd700 1 -- 172.21.15.79:0/3689930702 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17f800fd80 con 0x7f17fc0fcd00 2024-01-19T01:21:16.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.451+0000 7f17f2ffd700 1 -- 172.21.15.79:0/3689930702 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f800d310 con 0x7f17fc0fcd00 2024-01-19T01:21:16.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.452+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc115620 con 0x7f17fc0fcd00 2024-01-19T01:21:16.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.452+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc04bb80 con 0x7f17fc0fcd00 2024-01-19T01:21:16.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.453+0000 7f17f2ffd700 1 -- 172.21.15.79:0/3689930702 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f17f8010420 con 0x7f17fc0fcd00 2024-01-19T01:21:16.455 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.453+0000 7f17f2ffd700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f17e8068f70 0x7f17e806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:16.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.453+0000 7f17f2ffd700 1 -- 172.21.15.79:0/3689930702 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f17f8086d90 con 0x7f17fc0fcd00 2024-01-19T01:21:16.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.453+0000 7f1802283700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f17e8068f70 0x7f17e806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:16.456 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.454+0000 7f1802283700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f17e8068f70 0x7f17e806b420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009fd0 tx=0x7f17ec009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:16.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.457+0000 7f17f2ffd700 1 -- 172.21.15.79:0/3689930702 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f17f8053210 con 0x7f17fc0fcd00 2024-01-19T01:21:16.700 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:16.699+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17fc104510 con 0x7f17fc0fcd00 2024-01-19T01:21:16.700 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:standby 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: Dropping low affinity active daemon mds.cephfs.smithi079.cwngbt in favor of higher affinity standby. 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: Replacing daemon mds.cephfs.smithi079.cwngbt as rank 0 with standby daemon mds.cephfs.smithi079.qxcrcz 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 3 up:standby 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: osdmap e46: 6 total, 6 up, 6 in 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: fsmap cephfs:1/1 {0=cephfs.smithi079.qxcrcz=up:replay} 2 up:standby 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:16.701 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-01-19T01:21:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:standby 2024-01-19T01:21:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: Dropping low affinity active daemon mds.cephfs.smithi079.cwngbt in favor of higher affinity standby. 2024-01-19T01:21:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: Replacing daemon mds.cephfs.smithi079.cwngbt as rank 0 with standby daemon mds.cephfs.smithi079.qxcrcz 2024-01-19T01:21:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-01-19T01:21:16.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.cwngbt=up:active} 3 up:standby 2024-01-19T01:21:16.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: osdmap e46: 6 total, 6 up, 6 in 2024-01-19T01:21:16.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: fsmap cephfs:1/1 {0=cephfs.smithi079.qxcrcz=up:replay} 2 up:standby 2024-01-19T01:21:16.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:16.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:17.573 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.572+0000 7f17f2ffd700 1 -- 172.21.15.79:0/3689930702 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v14) v1 ==== 168+0+0 (secure 0 0 0) 0x7f17f805ae60 con 0x7f17fc0fcd00 2024-01-19T01:21:17.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f17e8068f70 msgr2=0x7f17e806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:17.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f17e8068f70 0x7f17e806b420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009fd0 tx=0x7f17ec009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:17.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fcd00 msgr2=0x7f17fc114c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:17.575 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fcd00 0x7f17fc114c50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f17f800b790 tx=0x7f17f8009250 comp rx=0 tx=0).stop 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 shutdown_connections 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f17e8068f70 0x7f17e806b420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17fc0fbaf0 0x7f17fc112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 --2- 172.21.15.79:0/3689930702 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f17fc0fcd00 0x7f17fc114c50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 >> 172.21.15.79:0/3689930702 conn(0x7f17fc0f7190 msgr2=0x7f17fc0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 shutdown_connections 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:17.574+0000 7f18044e7700 1 -- 172.21.15.79:0/3689930702 wait complete. 2024-01-19T01:21:17.576 INFO:teuthology.orchestra.run.smithi079.stderr:inline data enabled 2024-01-19T01:21:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:17 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:reconnect 2024-01-19T01:21:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:17 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6826/2748689961,v1:172.21.15.79:6827/2748689961] up:boot 2024-01-19T01:21:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:17 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] up:standby 2024-01-19T01:21:17.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:17 smithi152 ceph-mon[35694]: fsmap cephfs:1/1 {0=cephfs.smithi079.qxcrcz=up:reconnect} 3 up:standby 2024-01-19T01:21:17.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.cwngbt"}]: dispatch 2024-01-19T01:21:17.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:17 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3689930702' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-01-19T01:21:17.976 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:17 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:reconnect 2024-01-19T01:21:17.976 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:17 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6826/2748689961,v1:172.21.15.79:6827/2748689961] up:boot 2024-01-19T01:21:17.976 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:17 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] up:standby 2024-01-19T01:21:17.976 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:17 smithi079 ceph-mon[28374]: fsmap cephfs:1/1 {0=cephfs.smithi079.qxcrcz=up:reconnect} 3 up:standby 2024-01-19T01:21:17.976 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi079.cwngbt"}]: dispatch 2024-01-19T01:21:17.976 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:17 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3689930702' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-01-19T01:21:18.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:18 smithi079 ceph-mon[28374]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-01-19T01:21:18.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:18 smithi079 ceph-mon[28374]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-01-19T01:21:18.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:18 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:rejoin 2024-01-19T01:21:18.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:18 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3689930702' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-01-19T01:21:18.703 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:18 smithi079 ceph-mon[28374]: fsmap cephfs:1/1 {0=cephfs.smithi079.qxcrcz=up:rejoin} 3 up:standby 2024-01-19T01:21:18.704 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:18 smithi079 ceph-mon[28374]: daemon mds.cephfs.smithi079.qxcrcz is now active in filesystem cephfs as rank 0 2024-01-19T01:21:18.727 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-01-19T01:21:18.739 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:21:18.739 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph fs dump' 2024-01-19T01:21:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:18 smithi152 ceph-mon[35694]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-01-19T01:21:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:18 smithi152 ceph-mon[35694]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-01-19T01:21:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:18 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:rejoin 2024-01-19T01:21:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:18 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3689930702' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-01-19T01:21:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:18 smithi152 ceph-mon[35694]: fsmap cephfs:1/1 {0=cephfs.smithi079.qxcrcz=up:rejoin} 3 up:standby 2024-01-19T01:21:18.825 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:18 smithi152 ceph-mon[35694]: daemon mds.cephfs.smithi079.qxcrcz is now active in filesystem cephfs as rank 0 2024-01-19T01:21:19.080 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:19.836 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:19 smithi152 ceph-mon[35694]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-01-19T01:21:19.836 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:19 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:active 2024-01-19T01:21:19.836 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:19 smithi152 ceph-mon[35694]: mds.? [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] up:standby 2024-01-19T01:21:19.836 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:19 smithi152 ceph-mon[35694]: fsmap cephfs:1 {0=cephfs.smithi079.qxcrcz=up:active} 3 up:standby 2024-01-19T01:21:19.836 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:19.837 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:19.860 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:19 smithi079 ceph-mon[28374]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-01-19T01:21:19.860 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:19 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] up:active 2024-01-19T01:21:19.860 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:19 smithi079 ceph-mon[28374]: mds.? [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] up:standby 2024-01-19T01:21:19.860 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:19 smithi079 ceph-mon[28374]: fsmap cephfs:1 {0=cephfs.smithi079.qxcrcz=up:active} 3 up:standby 2024-01-19T01:21:19.860 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:19.861 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:20.891 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:20 smithi079 ceph-mon[28374]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 716 B/s wr, 3 op/s 2024-01-19T01:21:20.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:20 smithi152 ceph-mon[35694]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 716 B/s wr, 3 op/s 2024-01-19T01:21:21.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.543+0000 7f2d29b2c700 1 -- 172.21.15.79:0/662643266 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fbe10 msgr2=0x7f2d240fc280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:21.546 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.543+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/662643266 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fbe10 0x7f2d240fc280 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2d18004940 tx=0x7f2d18009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 -- 172.21.15.79:0/662643266 shutdown_connections 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/662643266 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fbe10 0x7f2d240fc280 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/662643266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d240fd190 0x7f2d240fd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 -- 172.21.15.79:0/662643266 >> 172.21.15.79:0/662643266 conn(0x7f2d240f6950 msgr2=0x7f2d240f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 -- 172.21.15.79:0/662643266 shutdown_connections 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 -- 172.21.15.79:0/662643266 wait complete. 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 Processor -- start 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 -- start start 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d240fbe10 0x7f2d2410c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fd190 0x7f2d2410c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2410cff0 con 0x7f2d240fd190 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.544+0000 7f2d29b2c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2410d890 con 0x7f2d240fbe10 2024-01-19T01:21:21.547 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d22ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fd190 0x7f2d2410c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d22ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fd190 0x7f2d2410c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34914/0 (socket says 172.21.15.79:34914) 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d22ffd700 1 -- 172.21.15.79:0/4188121153 learned_addr learned my addr 172.21.15.79:0/4188121153 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d237fe700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d240fbe10 0x7f2d2410c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d22ffd700 1 -- 172.21.15.79:0/4188121153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d240fbe10 msgr2=0x7f2d2410c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d22ffd700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d240fbe10 0x7f2d2410c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d22ffd700 1 -- 172.21.15.79:0/4188121153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18005040 con 0x7f2d240fd190 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d22ffd700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fd190 0x7f2d2410c9f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009380 tx=0x7f2d18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d0bfff700 1 -- 172.21.15.79:0/4188121153 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1800e040 con 0x7f2d240fd190 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2410db10 con 0x7f2d240fd190 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d0bfff700 1 -- 172.21.15.79:0/4188121153 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d18009af0 con 0x7f2d240fd190 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d0bfff700 1 -- 172.21.15.79:0/4188121153 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d18013660 con 0x7f2d240fd190 2024-01-19T01:21:21.548 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.545+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2410dff0 con 0x7f2d240fd190 2024-01-19T01:21:21.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.546+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2404bb80 con 0x7f2d240fd190 2024-01-19T01:21:21.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.547+0000 7f2d0bfff700 1 -- 172.21.15.79:0/4188121153 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2d1801e960 con 0x7f2d240fd190 2024-01-19T01:21:21.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.547+0000 7f2d0bfff700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2d0c068f60 0x7f2d0c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:21.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.547+0000 7f2d0bfff700 1 -- 172.21.15.79:0/4188121153 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2d1808ad20 con 0x7f2d240fd190 2024-01-19T01:21:21.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.547+0000 7f2d237fe700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2d0c068f60 0x7f2d0c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:21.549 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.548+0000 7f2d237fe700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2d0c068f60 0x7f2d0c06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004f40 tx=0x7f2d1400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:21.552 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.552+0000 7f2d0bfff700 1 -- 172.21.15.79:0/4188121153 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2d1805abb0 con 0x7f2d240fd190 2024-01-19T01:21:21.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.814+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2d24104550 con 0x7f2d240fd190 2024-01-19T01:21:21.815 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.814+0000 7f2d0bfff700 1 -- 172.21.15.79:0/4188121153 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1809 (secure 0 0 0) 0x7f2d1805a740 con 0x7f2d240fd190 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:e15 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 15 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-01-19T01:21:01.585027+0000 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-01-19T01:21:18.569403+0000 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-01-19T01:21:21.816 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 60 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 46 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 1 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:in 0 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:up {0=14472} 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-01-19T01:21:21.817 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data enabled 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 1 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.qxcrcz{0:14472} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:Standby daemons: 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.cwngbt{-1:14484} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.79:6826/2748689961,v1:172.21.15.79:6827/2748689961] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.hpxjws{-1:24255} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:21:21.818 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.btzecy{-1:24265} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2d0c068f60 msgr2=0x7f2d0c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2d0c068f60 0x7f2d0c06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d14004f40 tx=0x7f2d1400b040 comp rx=0 tx=0).stop 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fd190 msgr2=0x7f2d2410c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fd190 0x7f2d2410c9f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d18009380 tx=0x7f2d18009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 shutdown_connections 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2d0c068f60 0x7f2d0c06b410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2d240fbe10 0x7f2d2410c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:21.819 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 --2- 172.21.15.79:0/4188121153 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2d240fd190 0x7f2d2410c9f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:21.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 >> 172.21.15.79:0/4188121153 conn(0x7f2d240f6950 msgr2=0x7f2d24100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:21.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 shutdown_connections 2024-01-19T01:21:21.820 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:21.817+0000 7f2d29b2c700 1 -- 172.21.15.79:0/4188121153 wait complete. 2024-01-19T01:21:21.820 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 15 2024-01-19T01:21:22.444 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-01-19T01:21:22.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T01:21:22.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/4188121153' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:21:22.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:22.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:22.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:22.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:22.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:22.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:22 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-01-19T01:21:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/4188121153' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:21:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:22.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:22.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:22 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:23.349 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:24.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:24 smithi079 ceph-mon[28374]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:24.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:24 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:24.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:24 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:24.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:24 smithi152 ceph-mon[35694]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:24.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:24 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:24.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:24 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:25.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/175608503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fcd00 msgr2=0x7fc7a40fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:25.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/175608503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fcd00 0x7fc7a40fd150 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc78c004940 tx=0x7fc78c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:25.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/175608503 shutdown_connections 2024-01-19T01:21:25.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/175608503 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fcd00 0x7fc7a40fd150 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:25.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/175608503 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a40fbaf0 0x7fc7a40fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:25.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/175608503 >> 172.21.15.79:0/175608503 conn(0x7fc7a40f7190 msgr2=0x7fc7a40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/175608503 shutdown_connections 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.625+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/175608503 wait complete. 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a9cb6700 1 Processor -- start 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a9cb6700 1 -- start start 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a9cb6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fbaf0 0x7fc7a410c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a9cb6700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a40fcd00 0x7fc7a410c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a9cb6700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a410cfc0 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a9cb6700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a410d860 con 0x7fc7a40fcd00 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fbaf0 0x7fc7a410c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a37fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fbaf0 0x7fc7a410c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:34926/0 (socket says 172.21.15.79:34926) 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a37fe700 1 -- 172.21.15.79:0/1527882759 learned_addr learned my addr 172.21.15.79:0/1527882759 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc79affd700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a40fcd00 0x7fc7a410c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a37fe700 1 -- 172.21.15.79:0/1527882759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a40fcd00 msgr2=0x7fc7a410c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:25.629 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a37fe700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a40fcd00 0x7fc7a410c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:25.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.626+0000 7fc7a37fe700 1 -- 172.21.15.79:0/1527882759 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c005040 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.627+0000 7fc7a37fe700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fbaf0 0x7fc7a410c480 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc79400fce0 tx=0x7fc794004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:25.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.627+0000 7fc7a0ff9700 1 -- 172.21.15.79:0/1527882759 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc794007ea0 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.627+0000 7fc7a0ff9700 1 -- 172.21.15.79:0/1527882759 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc79400dcd0 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.627+0000 7fc7a0ff9700 1 -- 172.21.15.79:0/1527882759 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc794012460 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.627+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a410dae0 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.630 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.627+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a410e020 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.628+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a404bb80 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.628+0000 7fc7a0ff9700 1 -- 172.21.15.79:0/1527882759 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc79400d820 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.629+0000 7fc7a0ff9700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc790068f60 0x7fc79006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:25.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.629+0000 7fc79affd700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc790068f60 0x7fc79006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:25.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.629+0000 7fc7a0ff9700 1 -- 172.21.15.79:0/1527882759 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc794089470 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.631 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.629+0000 7fc79affd700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc790068f60 0x7fc79006b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b3b0 tx=0x7fc78c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:25.634 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.633+0000 7fc7a0ff9700 1 -- 172.21.15.79:0/1527882759 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc794059380 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.862+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7a41044b0 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.863 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.862+0000 7fc7a0ff9700 1 -- 172.21.15.79:0/1527882759 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4691 (secure 0 0 0) 0x7fc794058f10 con 0x7fc7a40fbaf0 2024-01-19T01:21:25.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc790068f60 msgr2=0x7fc79006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:25.865 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc790068f60 0x7fc79006b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b3b0 tx=0x7fc78c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fbaf0 msgr2=0x7fc7a410c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fbaf0 0x7fc7a410c480 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc79400fce0 tx=0x7fc794004600 comp rx=0 tx=0).stop 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 shutdown_connections 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc790068f60 0x7fc79006b410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc7a40fbaf0 0x7fc7a410c480 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 --2- 172.21.15.79:0/1527882759 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a40fcd00 0x7fc7a410c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.864+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 >> 172.21.15.79:0/1527882759 conn(0x7fc7a40f7190 msgr2=0x7fc7a40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.865+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 shutdown_connections 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:25.865+0000 7fc7a9cb6700 1 -- 172.21.15.79:0/1527882759 wait complete. 2024-01-19T01:21:25.866 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 15 2024-01-19T01:21:25.876 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-01-19T01:21:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:26 smithi152 ceph-mon[35694]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:26.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:26 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1527882759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T01:21:26.965 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:26 smithi079 ceph-mon[28374]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:26.965 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:26 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1527882759' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T01:21:26.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-01-19T01:21:28.459 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:28.907 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:28 smithi079 ceph-mon[28374]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:28.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:28 smithi152 ceph-mon[35694]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.4 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.111+0000 7fcc9062f700 1 -- 172.21.15.79:0/623427566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 msgr2=0x7fcc880ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.111+0000 7fcc9062f700 1 --2- 172.21.15.79:0/623427566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc880ffb10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcc84004940 tx=0x7fcc84009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.111+0000 7fcc9062f700 1 -- 172.21.15.79:0/623427566 shutdown_connections 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.111+0000 7fcc9062f700 1 --2- 172.21.15.79:0/623427566 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc88100050 0x7fcc88102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.111+0000 7fcc9062f700 1 --2- 172.21.15.79:0/623427566 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc880ffb10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.111+0000 7fcc9062f700 1 -- 172.21.15.79:0/623427566 >> 172.21.15.79:0/623427566 conn(0x7fcc880f7190 msgr2=0x7fcc880f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 -- 172.21.15.79:0/623427566 shutdown_connections 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 -- 172.21.15.79:0/623427566 wait complete. 2024-01-19T01:21:30.114 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 Processor -- start 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 -- start start 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc8810d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc88100050 0x7fcc8810da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8810e040 con 0x7fcc880fd730 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc9062f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8810e8e0 con 0x7fcc88100050 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc8e3cb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc8810d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc8e3cb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc8810d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48210/0 (socket says 172.21.15.79:48210) 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.112+0000 7fcc8e3cb700 1 -- 172.21.15.79:0/155150202 learned_addr learned my addr 172.21.15.79:0/155150202 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc8dbca700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc88100050 0x7fcc8810da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc8e3cb700 1 -- 172.21.15.79:0/155150202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc88100050 msgr2=0x7fcc8810da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc8e3cb700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc88100050 0x7fcc8810da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc8e3cb700 1 -- 172.21.15.79:0/155150202 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc84005040 con 0x7fcc880fd730 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc8e3cb700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc8810d500 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009360 tx=0x7fcc84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:30.115 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc7effd700 1 -- 172.21.15.79:0/155150202 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc8400e040 con 0x7fcc880fd730 2024-01-19T01:21:30.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8810eb60 con 0x7fcc880fd730 2024-01-19T01:21:30.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc7effd700 1 -- 172.21.15.79:0/155150202 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc8400bc50 con 0x7fcc880fd730 2024-01-19T01:21:30.116 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc7effd700 1 -- 172.21.15.79:0/155150202 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc8401c3f0 con 0x7fcc880fd730 2024-01-19T01:21:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.113+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8810f0a0 con 0x7fcc880fd730 2024-01-19T01:21:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.114+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8804bb80 con 0x7fcc880fd730 2024-01-19T01:21:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.114+0000 7fcc7effd700 1 -- 172.21.15.79:0/155150202 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcc84013420 con 0x7fcc880fd730 2024-01-19T01:21:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.115+0000 7fcc7effd700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc74068f60 0x7fcc7406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.115+0000 7fcc8dbca700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc74068f60 0x7fcc7406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.115+0000 7fcc7effd700 1 -- 172.21.15.79:0/155150202 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcc84088160 con 0x7fcc880fd730 2024-01-19T01:21:30.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.115+0000 7fcc8dbca700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc74068f60 0x7fcc7406b410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcc78004f40 tx=0x7fcc7800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:30.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.119+0000 7fcc7effd700 1 -- 172.21.15.79:0/155150202 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcc84057ff0 con 0x7fcc880fd730 2024-01-19T01:21:30.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.379+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fcc8810ecf0 con 0x7fcc880fd730 2024-01-19T01:21:30.380 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.379+0000 7fcc7effd700 1 -- 172.21.15.79:0/155150202 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+99 (secure 0 0 0) 0x7fcc8810ecf0 con 0x7fcc880fd730 2024-01-19T01:21:30.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.381+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc74068f60 msgr2=0x7fcc7406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:30.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.381+0000 7fcc9062f700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc74068f60 0x7fcc7406b410 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcc78004f40 tx=0x7fcc7800b040 comp rx=0 tx=0).stop 2024-01-19T01:21:30.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.381+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 msgr2=0x7fcc8810d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:30.382 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.381+0000 7fcc9062f700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc8810d500 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcc84009360 tx=0x7fcc84009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:30.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.382+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 shutdown_connections 2024-01-19T01:21:30.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.382+0000 7fcc9062f700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fcc74068f60 0x7fcc7406b410 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:30.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.382+0000 7fcc9062f700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fcc880fd730 0x7fcc8810d500 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:30.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.382+0000 7fcc9062f700 1 --2- 172.21.15.79:0/155150202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc88100050 0x7fcc8810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:30.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.382+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 >> 172.21.15.79:0/155150202 conn(0x7fcc880f7190 msgr2=0x7fcc880f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:30.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.382+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 shutdown_connections 2024-01-19T01:21:30.383 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:30.382+0000 7fcc9062f700 1 -- 172.21.15.79:0/155150202 wait complete. 2024-01-19T01:21:30.393 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-01-19T01:21:30.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:30 smithi152 ceph-mon[35694]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:30.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:30.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:30.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:30.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:30.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:30 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:30.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:30 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/155150202' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-01-19T01:21:30.934 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:30 smithi079 ceph-mon[28374]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 4 op/s 2024-01-19T01:21:30.934 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:30.934 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:30.934 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:30.934 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:30.935 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:30 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:30.935 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:30 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/155150202' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-01-19T01:21:30.958 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-01-19T01:21:30.970 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T01:21:31.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 -- 172.21.15.79:0/233159876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 msgr2=0x7f49b40fc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 --2- 172.21.15.79:0/233159876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b40fc500 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f49a00048d0 tx=0x7f49a0009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 -- 172.21.15.79:0/233159876 shutdown_connections 2024-01-19T01:21:31.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 --2- 172.21.15.79:0/233159876 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b40fc500 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 --2- 172.21.15.79:0/233159876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b40fae70 0x7f49b40fb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 -- 172.21.15.79:0/233159876 >> 172.21.15.79:0/233159876 conn(0x7f49b40f65c0 msgr2=0x7f49b40f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:31.083 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 -- 172.21.15.79:0/233159876 shutdown_connections 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.080+0000 7f49bb06e640 1 -- 172.21.15.79:0/233159876 wait complete. 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49bb06e640 1 Processor -- start 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49bb06e640 1 -- start start 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49bb06e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b40fae70 0x7f49b410c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49bb06e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b410cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49bb06e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b410d400 con 0x7f49b40fc090 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49bb06e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b4111dc0 con 0x7f49b40fae70 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b410cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b3fff640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b410cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48238/0 (socket says 172.21.15.79:48238) 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b3fff640 1 -- 172.21.15.79:0/869870058 learned_addr learned my addr 172.21.15.79:0/869870058 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b3fff640 1 -- 172.21.15.79:0/869870058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b40fae70 msgr2=0x7f49b410c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b3fff640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b40fae70 0x7f49b410c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b3fff640 1 -- 172.21.15.79:0/869870058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a0005040 con 0x7f49b40fc090 2024-01-19T01:21:31.084 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b3fff640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b410cd20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49a0003080 tx=0x7f49a0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.081+0000 7f49b17fa640 1 -- 172.21.15.79:0/869870058 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49a00094c0 con 0x7f49b40fc090 2024-01-19T01:21:31.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.082+0000 7f49b17fa640 1 -- 172.21.15.79:0/869870058 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49a0009620 con 0x7f49b40fc090 2024-01-19T01:21:31.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.082+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b4111fc0 con 0x7f49b40fc090 2024-01-19T01:21:31.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.082+0000 7f49b17fa640 1 -- 172.21.15.79:0/869870058 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49a003b640 con 0x7f49b40fc090 2024-01-19T01:21:31.085 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.082+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b4112450 con 0x7f49b40fc090 2024-01-19T01:21:31.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.083+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b41038d0 con 0x7f49b40fc090 2024-01-19T01:21:31.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.083+0000 7f49b17fa640 1 -- 172.21.15.79:0/869870058 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f49a002f070 con 0x7f49b40fc090 2024-01-19T01:21:31.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.083+0000 7f49b17fa640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f49a4068ed0 0x7f49a406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.084+0000 7f49b8e0a640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f49a4068ed0 0x7f49a406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.084+0000 7f49b17fa640 1 -- 172.21.15.79:0/869870058 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f49a00a94a0 con 0x7f49b40fc090 2024-01-19T01:21:31.086 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.084+0000 7f49b8e0a640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f49a4068ed0 0x7f49a406b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f49b40fbc00 tx=0x7f49a8006d20 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.088+0000 7f49b17fa640 1 -- 172.21.15.79:0/869870058 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f49a00758a0 con 0x7f49b40fc090 2024-01-19T01:21:31.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.251+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49b4112730 con 0x7f49b40fc090 2024-01-19T01:21:31.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.252+0000 7f49b17fa640 1 -- 172.21.15.79:0/869870058 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14406 (secure 0 0 0) 0x7f49a0075070 con 0x7f49b40fc090 2024-01-19T01:21:31.253 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:31.253 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":46,"fsid":"f3747e56-b667-11ee-95ae-87774f69a715","created":"2024-01-19T01:14:07.716096+0000","modified":"2024-01-19T01:21:15.558612+0000","last_up_change":"2024-01-19T01:19:43.531498+0000","last_in_change":"2024-01-19T01:19:25.050496+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T01:18:29.500877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-01-19T01:21:00.230430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-01-19T01:21:00.811797+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"4500f84c-0d7c-472b-8d18-35de388ec5bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6803","nonce":2692943028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6805","nonce":2692943028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6809","nonce":2692943028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":2692943028},{"type":"v1","addr":"172.21.15.79:6807","nonce":2692943028}]},"public_addr":"172.21.15.79:6803/2692943028","cluster_addr":"172.21.15.79:6805/2692943028","heartbeat_back_addr":"172.21.15.79:6809/2692943028","heartbeat_front_addr":"172.21.15.79:6807/2692943028","state":["exists","up"]},{"osd":1,"uuid":"0286a58d-91d9-48f3-9a73-cad13c064623","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6811","nonce":3254175144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6813","nonce":3254175144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6817","nonce":3254175144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":3254175144},{"type":"v1","addr":"172.21.15.79:6815","nonce":3254175144}]},"public_addr":"172.21.15.79:6811/3254175144","cluster_addr":"172.21.15.79:6813/3254175144","heartbeat_back_addr":"172.21.15.79:6817/3254175144","heartbeat_front_addr":"172.21.15.79:6815/3254175144","state":["exists","up"]},{"osd":2,"uuid":"1d32cd7f-64af-4edf-b9cf-3db2519b9bd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6819","nonce":3125547025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6821","nonce":3125547025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6825","nonce":3125547025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":3125547025},{"type":"v1","addr":"172.21.15.79:6823","nonce":3125547025}]},"public_addr":"172.21.15.79:6819/3125547025","cluster_addr":"172.21.15.79:6821/3125547025","heartbeat_back_addr":"172.21.15.79:6825/3125547025","heartbeat_front_addr":"172.21.15.79:6823/3125547025","state":["exists","up"]},{"osd":3,"uuid":"4ac48076-9143-445e-bac1-0ae801b0d3fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6801","nonce":2401180651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6803","nonce":2401180651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6807","nonce":2401180651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2401180651},{"type":"v1","addr":"172.21.15.152:6805","nonce":2401180651}]},"public_addr":"172.21.15.152:6801/2401180651","cluster_addr":"172.21.15.152:6803/2401180651","heartbeat_back_addr":"172.21.15.152:6807/2401180651","heartbeat_front_addr":"172.21.15.152:6805/2401180651","state":["exists","up"]},{"osd":4,"uuid":"7c505c3b-0853-4af4-bb1b-4e32ba16bede","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6809","nonce":1289083445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6811","nonce":1289083445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6815","nonce":1289083445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1289083445},{"type":"v1","addr":"172.21.15.152:6813","nonce":1289083445}]},"public_addr":"172.21.15.152:6809/1289083445","cluster_addr":"172.21.15.152:6811/1289083445","heartbeat_back_addr":"172.21.15.152:6815/1289083445","heartbeat_front_addr":"172.21.15.152:6813/1289083445","state":["exists","up"]},{"osd":5,"uuid":"c343c013-2f74-4c20-83bd-14e41c304d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6817","nonce":3123617721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6819","nonce":3123617721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6823","nonce":3123617721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3123617721},{"type":"v1","addr":"172.21.15.152:6821","nonce":3123617721}]},"public_addr":"172.21.15.152:6817/3123617721","cluster_addr":"172.21.15.152:6819/3123617721","heartbeat_back_addr":"172.21.15.152:6823/3123617721","heartbeat_front_addr":"172.21.15.152:6821/3123617721","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:17:31.789908+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:00.346643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:27.657969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:18:51.195401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:17.092535+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-01-19T01:19:42.162295+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:6827/1029843807":"2024-01-20T01:21:15.558589+0000","172.21.15.79:0/547589622":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/2783373766":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/907783696":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/4252083415":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/2671943691":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6800/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6800/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:6801/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/1174599613":"2024-01-20T01:15:09.094050+0000","172.21.15.79:0/207141703":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6801/3074418411":"2024-01-20T01:15:09.094050+0000","172.21.15.79:6801/430351410":"2024-01-20T01:14:38.242827+0000","172.21.15.79:0/780174951":"2024-01-20T01:16:19.162905+0000","172.21.15.79:0/1393070546":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6800/4076006582":"2024-01-20T01:16:19.162905+0000","172.21.15.79:6826/1029843807":"2024-01-20T01:21:15.558589+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T01:21:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f49a4068ed0 msgr2=0x7f49a406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f49a4068ed0 0x7f49a406b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f49b40fbc00 tx=0x7f49a8006d20 comp rx=0 tx=0).stop 2024-01-19T01:21:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 msgr2=0x7f49b410cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b410cd20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49a0003080 tx=0x7f49a0009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 shutdown_connections 2024-01-19T01:21:31.255 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f49b40fc090 0x7f49b410cd20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f49a4068ed0 0x7f49a406b380 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 --2- 172.21.15.79:0/869870058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f49b40fae70 0x7f49b410c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 >> 172.21.15.79:0/869870058 conn(0x7f49b40f65c0 msgr2=0x7f49b40653c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 shutdown_connections 2024-01-19T01:21:31.256 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.254+0000 7f49bb06e640 1 -- 172.21.15.79:0/869870058 wait complete. 2024-01-19T01:21:31.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-19T01:18:29.500877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-01-19T01:21:00.230430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-01-19T01:21:00.811797+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-19T01:21:31.267 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-19T01:21:31.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 -- 172.21.15.79:0/442712200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 msgr2=0x7f8b340fc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 --2- 172.21.15.79:0/442712200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b340fc4c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8b240048d0 tx=0x7f8b24009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 -- 172.21.15.79:0/442712200 shutdown_connections 2024-01-19T01:21:31.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 --2- 172.21.15.79:0/442712200 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b340fc4c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.369 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 --2- 172.21.15.79:0/442712200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b340fae30 0x7f8b340fb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 -- 172.21.15.79:0/442712200 >> 172.21.15.79:0/442712200 conn(0x7f8b340f6580 msgr2=0x7f8b340f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 -- 172.21.15.79:0/442712200 shutdown_connections 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.366+0000 7f8b3c425640 1 -- 172.21.15.79:0/442712200 wait complete. 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3c425640 1 Processor -- start 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3c425640 1 -- start start 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3c425640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b340fae30 0x7f8b3410c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3c425640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b3410ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3c425640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3410d300 con 0x7f8b340fc050 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3c425640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34111d50 con 0x7f8b340fae30 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b399c0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b3410ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b399c0640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b3410ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48260/0 (socket says 172.21.15.79:48260) 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b399c0640 1 -- 172.21.15.79:0/1215605282 learned_addr learned my addr 172.21.15.79:0/1215605282 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3a1c1640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b340fae30 0x7f8b3410c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.370 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b399c0640 1 -- 172.21.15.79:0/1215605282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b340fae30 msgr2=0x7f8b3410c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b399c0640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b340fae30 0x7f8b3410c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b399c0640 1 -- 172.21.15.79:0/1215605282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b24005040 con 0x7f8b340fc050 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.367+0000 7f8b3a1c1640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b340fae30 0x7f8b3410c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.368+0000 7f8b399c0640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b3410ccb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8b2402bda0 tx=0x7f8b24009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.368+0000 7f8b2affd640 1 -- 172.21.15.79:0/1215605282 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b240094c0 con 0x7f8b340fc050 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.368+0000 7f8b2affd640 1 -- 172.21.15.79:0/1215605282 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b24009620 con 0x7f8b340fc050 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.368+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b34111e90 con 0x7f8b340fc050 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.368+0000 7f8b2affd640 1 -- 172.21.15.79:0/1215605282 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b2403b400 con 0x7f8b340fc050 2024-01-19T01:21:31.371 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.368+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b34112400 con 0x7f8b340fc050 2024-01-19T01:21:31.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.369+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b341038a0 con 0x7f8b340fc050 2024-01-19T01:21:31.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.369+0000 7f8b2affd640 1 -- 172.21.15.79:0/1215605282 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8b2402f070 con 0x7f8b340fc050 2024-01-19T01:21:31.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.369+0000 7f8b2affd640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8b20068ed0 0x7f8b2006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.370+0000 7f8b2affd640 1 -- 172.21.15.79:0/1215605282 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8b240a9480 con 0x7f8b340fc050 2024-01-19T01:21:31.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.370+0000 7f8b3a1c1640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8b20068ed0 0x7f8b2006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.370+0000 7f8b3a1c1640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8b20068ed0 0x7f8b2006b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8b340fbbc0 tx=0x7f8b30004d00 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.374 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.374+0000 7f8b2affd640 1 -- 172.21.15.79:0/1215605282 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8b24075880 con 0x7f8b340fc050 2024-01-19T01:21:31.539 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.538+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b34112810 con 0x7f8b340fc050 2024-01-19T01:21:31.540 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.539+0000 7f8b2affd640 1 -- 172.21.15.79:0/1215605282 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b24078ea0 con 0x7f8b340fc050 2024-01-19T01:21:31.540 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.540+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8b20068ed0 msgr2=0x7f8b2006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.540+0000 7f8b3c425640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8b20068ed0 0x7f8b2006b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8b340fbbc0 tx=0x7f8b30004d00 comp rx=0 tx=0).stop 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.540+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 msgr2=0x7f8b3410ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.540+0000 7f8b3c425640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b3410ccb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8b2402bda0 tx=0x7f8b24009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.541+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 shutdown_connections 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.541+0000 7f8b3c425640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f8b340fc050 0x7f8b3410ccb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.541+0000 7f8b3c425640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f8b20068ed0 0x7f8b2006b380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.541+0000 7f8b3c425640 1 --2- 172.21.15.79:0/1215605282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b340fae30 0x7f8b3410c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.541+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 >> 172.21.15.79:0/1215605282 conn(0x7f8b340f6580 msgr2=0x7f8b34064130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.541+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 shutdown_connections 2024-01-19T01:21:31.542 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.541+0000 7f8b3c425640 1 -- 172.21.15.79:0/1215605282 wait complete. 2024-01-19T01:21:31.552 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-01-19T01:21:31.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.653+0000 7f0c3079d640 1 -- 172.21.15.79:0/1429931855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fae30 msgr2=0x7f0c280fb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.656 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.653+0000 7f0c3079d640 1 --2- 172.21.15.79:0/1429931855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fae30 0x7f0c280fb240 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0c240048d0 tx=0x7f0c24009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 -- 172.21.15.79:0/1429931855 shutdown_connections 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 --2- 172.21.15.79:0/1429931855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c280fc050 0x7f0c280fc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 --2- 172.21.15.79:0/1429931855 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fae30 0x7f0c280fb240 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 -- 172.21.15.79:0/1429931855 >> 172.21.15.79:0/1429931855 conn(0x7f0c280f6580 msgr2=0x7f0c280f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 -- 172.21.15.79:0/1429931855 shutdown_connections 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 -- 172.21.15.79:0/1429931855 wait complete. 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 Processor -- start 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 -- start start 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c280fae30 0x7f0c2810b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fc050 0x7f0c2810bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2810c1e0 con 0x7f0c280fc050 2024-01-19T01:21:31.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c3079d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28110c30 con 0x7f0c280fae30 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c2dd38640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fc050 0x7f0c2810bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c2dd38640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fc050 0x7f0c2810bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48278/0 (socket says 172.21.15.79:48278) 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c2dd38640 1 -- 172.21.15.79:0/3227680585 learned_addr learned my addr 172.21.15.79:0/3227680585 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.654+0000 7f0c2e539640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c280fae30 0x7f0c2810b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c2dd38640 1 -- 172.21.15.79:0/3227680585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c280fae30 msgr2=0x7f0c2810b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c2dd38640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c280fae30 0x7f0c2810b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c2dd38640 1 -- 172.21.15.79:0/3227680585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c24005040 con 0x7f0c280fc050 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c2dd38640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fc050 0x7f0c2810bb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800fce0 tx=0x7f0c18004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c1effd640 1 -- 172.21.15.79:0/3227680585 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c18007a20 con 0x7f0c280fc050 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c1effd640 1 -- 172.21.15.79:0/3227680585 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c18007b80 con 0x7f0c280fc050 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c28064e10 con 0x7f0c280fc050 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c1effd640 1 -- 172.21.15.79:0/3227680585 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c180126a0 con 0x7f0c280fc050 2024-01-19T01:21:31.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.655+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c280653b0 con 0x7f0c280fc050 2024-01-19T01:21:31.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.656+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c281038a0 con 0x7f0c280fc050 2024-01-19T01:21:31.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.656+0000 7f0c1effd640 1 -- 172.21.15.79:0/3227680585 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0c18007cf0 con 0x7f0c280fc050 2024-01-19T01:21:31.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.657+0000 7f0c1effd640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c14068ed0 0x7f0c1406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.657+0000 7f0c1effd640 1 -- 172.21.15.79:0/3227680585 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0c18089330 con 0x7f0c280fc050 2024-01-19T01:21:31.659 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.657+0000 7f0c2e539640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c14068ed0 0x7f0c1406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.660 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.657+0000 7f0c2e539640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c14068ed0 0x7f0c1406b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c24000c00 tx=0x7f0c24009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.662 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.661+0000 7f0c1effd640 1 -- 172.21.15.79:0/3227680585 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0c18055860 con 0x7f0c280fc050 2024-01-19T01:21:31.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.822+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f0c280658d0 con 0x7f0c280fc050 2024-01-19T01:21:31.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.823+0000 7f0c1effd640 1 -- 172.21.15.79:0/3227680585 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0c18058e80 con 0x7f0c280fc050 2024-01-19T01:21:31.824 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c14068ed0 msgr2=0x7f0c1406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c14068ed0 0x7f0c1406b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c24000c00 tx=0x7f0c24009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fc050 msgr2=0x7f0c2810bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fc050 0x7f0c2810bb90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0c1800fce0 tx=0x7f0c18004600 comp rx=0 tx=0).stop 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 shutdown_connections 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c280fc050 0x7f0c2810bb90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c14068ed0 0x7f0c1406b380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 --2- 172.21.15.79:0/3227680585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c280fae30 0x7f0c2810b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 >> 172.21.15.79:0/3227680585 conn(0x7f0c280f6580 msgr2=0x7f0c280ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 shutdown_connections 2024-01-19T01:21:31.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.825+0000 7f0c3079d640 1 -- 172.21.15.79:0/3227680585 wait complete. 2024-01-19T01:21:31.835 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:31 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/869870058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:21:31.835 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:31 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1215605282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T01:21:31.836 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-01-19T01:21:31.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:31 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/869870058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-19T01:21:31.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:31 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1215605282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.934+0000 7f42abb5d640 1 -- 172.21.15.79:0/2652272301 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 msgr2=0x7f42a40fda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.934+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2652272301 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a40fda70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f429c009c50 tx=0x7f429c004d10 comp rx=0 tx=0).stop 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 -- 172.21.15.79:0/2652272301 shutdown_connections 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2652272301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42a40fdfb0 0x7f42a40fa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2652272301 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a40fda70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 -- 172.21.15.79:0/2652272301 >> 172.21.15.79:0/2652272301 conn(0x7f42a40f5da0 msgr2=0x7f42a40f81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 -- 172.21.15.79:0/2652272301 shutdown_connections 2024-01-19T01:21:31.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 -- 172.21.15.79:0/2652272301 wait complete. 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 Processor -- start 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 -- start start 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a410c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42a40fdfb0 0x7f42a410cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a410d380 con 0x7f42a40fd660 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42abb5d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4111dd0 con 0x7f42a40fdfb0 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42a98f9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a410c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42a98f9640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a410c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48296/0 (socket says 172.21.15.79:48296) 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.935+0000 7f42a98f9640 1 -- 172.21.15.79:0/2602722942 learned_addr learned my addr 172.21.15.79:0/2602722942 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f42a98f9640 1 -- 172.21.15.79:0/2602722942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42a40fdfb0 msgr2=0x7f42a410cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f42a98f9640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42a40fdfb0 0x7f42a410cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f42a98f9640 1 -- 172.21.15.79:0/2602722942 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c0098b0 con 0x7f42a40fd660 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f42a98f9640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a410c7f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f429c02c920 tx=0x7f429c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f429a7fc640 1 -- 172.21.15.79:0/2602722942 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f429c03a070 con 0x7f42a40fd660 2024-01-19T01:21:31.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f429a7fc640 1 -- 172.21.15.79:0/2602722942 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f429c009690 con 0x7f42a40fd660 2024-01-19T01:21:31.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a4111f10 con 0x7f42a40fd660 2024-01-19T01:21:31.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f429a7fc640 1 -- 172.21.15.79:0/2602722942 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f429c03e720 con 0x7f42a40fd660 2024-01-19T01:21:31.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.936+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a4112480 con 0x7f42a40fd660 2024-01-19T01:21:31.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.937+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a41049e0 con 0x7f42a40fd660 2024-01-19T01:21:31.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.937+0000 7f429a7fc640 1 -- 172.21.15.79:0/2602722942 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f429c032070 con 0x7f42a40fd660 2024-01-19T01:21:31.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.938+0000 7f429a7fc640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4290068ed0 0x7f429006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:31.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.938+0000 7f429a7fc640 1 -- 172.21.15.79:0/2602722942 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f429c0a8e00 con 0x7f42a40fd660 2024-01-19T01:21:31.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.938+0000 7f42a90f8640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4290068ed0 0x7f429006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:31.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.938+0000 7f42a90f8640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4290068ed0 0x7f429006b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f42a410dd40 tx=0x7f4294009c50 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:31.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:31.942+0000 7f429a7fc640 1 -- 172.21.15.79:0/2602722942 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f429c075200 con 0x7f42a40fd660 2024-01-19T01:21:32.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.104+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f42a4112bc0 con 0x7f42a40fd660 2024-01-19T01:21:32.105 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.105+0000 7f429a7fc640 1 -- 172.21.15.79:0/2602722942 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f429c078820 con 0x7f42a40fd660 2024-01-19T01:21:32.106 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 32 2024-01-19T01:21:32.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4290068ed0 msgr2=0x7f429006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:32.107 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4290068ed0 0x7f429006b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f42a410dd40 tx=0x7f4294009c50 comp rx=0 tx=0).stop 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 msgr2=0x7f42a410c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a410c7f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f429c02c920 tx=0x7f429c004d10 comp rx=0 tx=0).stop 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 shutdown_connections 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4290068ed0 0x7f429006b380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42a40fdfb0 0x7f42a410cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 --2- 172.21.15.79:0/2602722942 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f42a40fd660 0x7f42a410c7f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 >> 172.21.15.79:0/2602722942 conn(0x7f42a40f5da0 msgr2=0x7f42a4100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 shutdown_connections 2024-01-19T01:21:32.108 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.106+0000 7f42abb5d640 1 -- 172.21.15.79:0/2602722942 wait complete. 2024-01-19T01:21:32.118 DEBUG:teuthology.orchestra.run.smithi079:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-19T01:21:32.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 -- 172.21.15.79:0/473424531 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0f9fb0 msgr2=0x7fc08c0fa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:32.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 --2- 172.21.15.79:0/473424531 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0f9fb0 0x7fc08c0fa420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0048d0 tx=0x7fc07c009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 -- 172.21.15.79:0/473424531 shutdown_connections 2024-01-19T01:21:32.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 --2- 172.21.15.79:0/473424531 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0f9fb0 0x7fc08c0fa420 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 --2- 172.21.15.79:0/473424531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc08c0fb2d0 0x7fc08c0fb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.216 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 -- 172.21.15.79:0/473424531 >> 172.21.15.79:0/473424531 conn(0x7fc08c0f5d00 msgr2=0x7fc08c0f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 -- 172.21.15.79:0/473424531 shutdown_connections 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.213+0000 7fc093a6e640 1 -- 172.21.15.79:0/473424531 wait complete. 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc093a6e640 1 Processor -- start 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc093a6e640 1 -- start start 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc093a6e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc08c0f9fb0 0x7fc08c112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc093a6e640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0fb2d0 0x7fc08c113260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc093a6e640 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c1138b0 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc093a6e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c118300 con 0x7fc08c0f9fb0 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc091009640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0fb2d0 0x7fc08c113260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc091009640 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0fb2d0 0x7fc08c113260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48318/0 (socket says 172.21.15.79:48318) 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc091009640 1 -- 172.21.15.79:0/3228228178 learned_addr learned my addr 172.21.15.79:0/3228228178 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc09180a640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc08c0f9fb0 0x7fc08c112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc091009640 1 -- 172.21.15.79:0/3228228178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc08c0f9fb0 msgr2=0x7fc08c112d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc091009640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc08c0f9fb0 0x7fc08c112d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc091009640 1 -- 172.21.15.79:0/3228228178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c005040 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.214+0000 7fc091009640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0fb2d0 0x7fc08c113260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc07c034040 tx=0x7fc07c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:32.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.215+0000 7fc0827fc640 1 -- 172.21.15.79:0/3228228178 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc07c0094c0 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.215+0000 7fc0827fc640 1 -- 172.21.15.79:0/3228228178 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc07c009620 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.215+0000 7fc0827fc640 1 -- 172.21.15.79:0/3228228178 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc07c03b640 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.215+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c118440 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.215+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c1189b0 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.218 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.215+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c10af80 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.216+0000 7fc0827fc640 1 -- 172.21.15.79:0/3228228178 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc07c02f070 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.216+0000 7fc0827fc640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc078068ed0 0x7fc07806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:32.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.216+0000 7fc0827fc640 1 -- 172.21.15.79:0/3228228178 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc07c0a9500 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.216+0000 7fc09180a640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc078068ed0 0x7fc07806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:32.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.217+0000 7fc09180a640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc078068ed0 0x7fc07806b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc088004bc0 tx=0x7fc088009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:32.221 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.220+0000 7fc0827fc640 1 -- 172.21.15.79:0/3228228178 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc07c075900 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.419+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc08c118d60 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.419+0000 7fc0827fc640 1 -- 172.21.15.79:0/3228228178 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4691 (secure 0 0 0) 0x7fc07c078f20 con 0x7fc08c0fb2d0 2024-01-19T01:21:32.421 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:21:32.421 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"cephfs.smithi079.cwngbt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.79:6827/2748689961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":2748689961},{"type":"v1","addr":"172.21.15.79:6827","nonce":2748689961}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24255,"name":"cephfs.smithi152.hpxjws","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.152:6825/394022976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":394022976},{"type":"v1","addr":"172.21.15.152:6825","nonce":394022976}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24265,"name":"cephfs.smithi152.btzecy","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.152:6827/2478066701","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2478066701},{"type":"v1","addr":"172.21.15.152:6827","nonce":2478066701}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-19T01:21:01.585027+0000","modified":"2024-01-19T01:21:18.569403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14472":{"gid":14472,"name":"cephfs.smithi079.qxcrcz","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.79:6829/2769248432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":2769248432},{"type":"v1","addr":"172.21.15.79:6829","nonce":2769248432}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-01-19T01:21:32.421 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 15 2024-01-19T01:21:32.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc078068ed0 msgr2=0x7fc07806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:32.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc078068ed0 0x7fc07806b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc088004bc0 tx=0x7fc088009040 comp rx=0 tx=0).stop 2024-01-19T01:21:32.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0fb2d0 msgr2=0x7fc08c113260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:32.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0fb2d0 0x7fc08c113260 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc07c034040 tx=0x7fc07c009da0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.422 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 shutdown_connections 2024-01-19T01:21:32.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fc08c0fb2d0 0x7fc08c113260 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fc078068ed0 0x7fc07806b380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 --2- 172.21.15.79:0/3228228178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc08c0f9fb0 0x7fc08c112d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:32.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 >> 172.21.15.79:0/3228228178 conn(0x7fc08c0f5d00 msgr2=0x7fc08c0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:32.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 shutdown_connections 2024-01-19T01:21:32.423 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:32.421+0000 7fc093a6e640 1 -- 172.21.15.79:0/3228228178 wait complete. 2024-01-19T01:21:32.433 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2024-01-19T01:21:32.433 INFO:teuthology.run_tasks:Running task kclient... 2024-01-19T01:21:32.456 INFO:tasks.kclient:Mounting kernel clients... 2024-01-19T01:21:32.457 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-01-19T01:21:32.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:32.457 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-01-19T01:21:32.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:32.473 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link delete ceph-brx 2024-01-19T01:21:32.541 INFO:teuthology.orchestra.run.smithi079.stderr:Cannot find device "ceph-brx" 2024-01-19T01:21:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T01:21:32.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:32.543 DEBUG:teuthology.orchestra.run.smithi152:> ip netns list 2024-01-19T01:21:32.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:32.562 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link delete ceph-brx 2024-01-19T01:21:32.638 INFO:teuthology.orchestra.run.smithi152.stderr:Cannot find device "ceph-brx" 2024-01-19T01:21:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T01:21:32.640 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-01-19T01:21:32.640 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-19T01:21:32.640 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:32.640 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph fs ls 2024-01-19T01:21:32.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:32 smithi079 ceph-mon[28374]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-01-19T01:21:32.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:32 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3227680585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-01-19T01:21:32.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:32 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2602722942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-01-19T01:21:32.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:32 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3228228178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T01:21:32.895 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:32.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:32 smithi152 ceph-mon[35694]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-01-19T01:21:32.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:32 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3227680585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-01-19T01:21:32.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:32 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2602722942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-01-19T01:21:32.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:32 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3228228178' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/517439319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 msgr2=0x7f2c240fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/517439319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c240fbfb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c20004940 tx=0x7f2c20009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/517439319 shutdown_connections 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/517439319 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c240fbfb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/517439319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24101b00 0x7f2c24101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/517439319 >> 172.21.15.79:0/517439319 conn(0x7f2c240719b0 msgr2=0x7f2c24071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/517439319 shutdown_connections 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.574+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/517439319 wait complete. 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2c5a1700 1 Processor -- start 2024-01-19T01:21:34.577 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2c5a1700 1 -- start start 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2c5a1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c2410c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2c5a1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24101b00 0x7f2c2410c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2c5a1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2410ce50 con 0x7f2c240fbb40 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2c5a1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2410cf90 con 0x7f2c24101b00 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2a33d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c2410c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2a33d700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c2410c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:48338/0 (socket says 172.21.15.79:48338) 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2a33d700 1 -- 172.21.15.79:0/1799941209 learned_addr learned my addr 172.21.15.79:0/1799941209 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c29b3c700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24101b00 0x7f2c2410c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2a33d700 1 -- 172.21.15.79:0/1799941209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24101b00 msgr2=0x7f2c2410c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2a33d700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24101b00 0x7f2c2410c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.575+0000 7f2c2a33d700 1 -- 172.21.15.79:0/1799941209 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20005040 con 0x7f2c240fbb40 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.576+0000 7f2c2a33d700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c2410c3d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c830 tx=0x7f2c14004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.576+0000 7f2c1affd700 1 -- 172.21.15.79:0/1799941209 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1400a760 con 0x7f2c240fbb40 2024-01-19T01:21:34.578 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.576+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c24119a40 con 0x7f2c240fbb40 2024-01-19T01:21:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.576+0000 7f2c1affd700 1 -- 172.21.15.79:0/1799941209 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c1400ad70 con 0x7f2c240fbb40 2024-01-19T01:21:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.576+0000 7f2c1affd700 1 -- 172.21.15.79:0/1799941209 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c140113a0 con 0x7f2c240fbb40 2024-01-19T01:21:34.579 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.576+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c24119e40 con 0x7f2c240fbb40 2024-01-19T01:21:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.577+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c24103fb0 con 0x7f2c240fbb40 2024-01-19T01:21:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.577+0000 7f2c1affd700 1 -- 172.21.15.79:0/1799941209 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2c14012400 con 0x7f2c240fbb40 2024-01-19T01:21:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.577+0000 7f2c1affd700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2c10068f60 0x7f2c1006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.577+0000 7f2c1affd700 1 -- 172.21.15.79:0/1799941209 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2c14086e60 con 0x7f2c240fbb40 2024-01-19T01:21:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.578+0000 7f2c29b3c700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2c10068f60 0x7f2c1006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:34.580 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.578+0000 7f2c29b3c700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2c10068f60 0x7f2c1006b410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c20004f90 tx=0x7f2c20009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:34.582 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.582+0000 7f2c1affd700 1 -- 172.21.15.79:0/1799941209 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2c140532e0 con 0x7f2c240fbb40 2024-01-19T01:21:34.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.810+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2c24065b70 con 0x7f2c240fbb40 2024-01-19T01:21:34.811 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.810+0000 7f2c1affd700 1 -- 172.21.15.79:0/1799941209 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f2c14056900 con 0x7f2c240fbb40 2024-01-19T01:21:34.811 INFO:teuthology.orchestra.run.smithi079.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-01-19T01:21:34.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2c10068f60 msgr2=0x7f2c1006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:34.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2c10068f60 0x7f2c1006b410 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2c20004f90 tx=0x7f2c20009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:34.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 msgr2=0x7f2c2410c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:34.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c2410c3d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c830 tx=0x7f2c14004470 comp rx=0 tx=0).stop 2024-01-19T01:21:34.813 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 shutdown_connections 2024-01-19T01:21:34.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f2c10068f60 0x7f2c1006b410 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:34.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f2c240fbb40 0x7f2c2410c3d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:34.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 --2- 172.21.15.79:0/1799941209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c24101b00 0x7f2c2410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:34.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 >> 172.21.15.79:0/1799941209 conn(0x7f2c240719b0 msgr2=0x7f2c240fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:34.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 shutdown_connections 2024-01-19T01:21:34.814 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:34.812+0000 7f2c2c5a1700 1 -- 172.21.15.79:0/1799941209 wait complete. 2024-01-19T01:21:34.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:34 smithi152 ceph-mon[35694]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-01-19T01:21:34.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:34 smithi079 ceph-mon[28374]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-01-19T01:21:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi079.front.sepia.ceph.com 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-19T01:21:35.592 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-19T01:21:35.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:35.593 DEBUG:teuthology.orchestra.run.smithi079:> ip addr 2024-01-19T01:21:35.609 INFO:teuthology.orchestra.run.smithi079.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-19T01:21:35.609 INFO:teuthology.orchestra.run.smithi079.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: inet 127.0.0.1/8 scope host lo 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: inet6 ::1/128 scope host 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:d8:a1:6a brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: altname enp7s0f0 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:d8:a1:6b brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: altname enp7s0f1 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:8f:ce:86 brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: link/ether 0c:c4:7a:8f:ce:87 brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:35.610 INFO:teuthology.orchestra.run.smithi079.stdout: inet 172.21.15.79/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-19T01:21:35.611 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft 42276sec preferred_lft 42276sec 2024-01-19T01:21:35.611 INFO:teuthology.orchestra.run.smithi079.stdout: inet6 fe80::ec4:7aff:fe8f:ce87/64 scope link 2024-01-19T01:21:35.611 INFO:teuthology.orchestra.run.smithi079.stdout: valid_lft forever preferred_lft forever 2024-01-19T01:21:35.611 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-19T01:21:35.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:35.611 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec bash -c ' 2024-01-19T01:21:35.611 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-01-19T01:21:35.611 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link add name ceph-brx type bridge 2024-01-19T01:21:35.612 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip addr flush dev ceph-brx 2024-01-19T01:21:35.612 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set ceph-brx up 2024-01-19T01:21:35.612 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-19T01:21:35.612 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-01-19T01:21:35.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:35.795 DEBUG:teuthology.orchestra.run.smithi079:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-19T01:21:35.820 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:35 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/1799941209' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T01:21:35.823 INFO:teuthology.orchestra.run.smithi079.stdout:1 2024-01-19T01:21:35.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:35.825 DEBUG:teuthology.orchestra.run.smithi079:> route 2024-01-19T01:21:35.890 INFO:teuthology.orchestra.run.smithi079.stdout:Kernel IP routing table 2024-01-19T01:21:35.891 INFO:teuthology.orchestra.run.smithi079.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-19T01:21:35.891 INFO:teuthology.orchestra.run.smithi079.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-19T01:21:35.891 INFO:teuthology.orchestra.run.smithi079.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-19T01:21:35.891 INFO:teuthology.orchestra.run.smithi079.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-19T01:21:35.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:35.891 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec bash -c ' 2024-01-19T01:21:35.891 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-01-19T01:21:35.891 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-19T01:21:35.892 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-19T01:21:35.892 DEBUG:teuthology.orchestra.run.smithi079:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-19T01:21:35.892 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-01-19T01:21:35.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:35 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/1799941209' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T01:21:36.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.194 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list 2024-01-19T01:21:36.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.209 DEBUG:teuthology.orchestra.run.smithi079:> ip netns list-id 2024-01-19T01:21:36.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.264 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec bash -c ' 2024-01-19T01:21:36.264 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-01-19T01:21:36.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-19T01:21:36.264 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-19T01:21:36.264 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-01-19T01:21:36.348 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-19T01:21:36.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec bash -c ' 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-19T01:21:36.349 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-01-19T01:21:36.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.450 DEBUG:teuthology.orchestra.run.smithi079:> (cd / && exec bash -c ' 2024-01-19T01:21:36.450 DEBUG:teuthology.orchestra.run.smithi079:> set -e 2024-01-19T01:21:36.450 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set brx.0 up 2024-01-19T01:21:36.450 DEBUG:teuthology.orchestra.run.smithi079:> sudo ip link set dev brx.0 master ceph-brx 2024-01-19T01:21:36.450 DEBUG:teuthology.orchestra.run.smithi079:> ') 2024-01-19T01:21:36.493 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T01:21:36.493 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:36.548 INFO:teuthology.orchestra.run.smithi079.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-19T01:21:36.549 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T01:21:36.549 DEBUG:teuthology.orchestra.run.smithi079:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:36.603 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-19T01:21:36.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.604 DEBUG:teuthology.orchestra.run.smithi079:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T01:21:36.709 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T01:21:36.710 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-19T01:21:36.710 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-01-19T01:21:36.825 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T01:21:36.825 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-19T01:21:36.825 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-01-19T01:21:36.825 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.152:3300" 2024-01-19T01:21:36.825 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T01:21:36.830 INFO:teuthology.orchestra.run.smithi079.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-01-19T01:21:36.830 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-01-19T01:21:36.830 INFO:teuthology.orchestra.run.smithi079.stdout:invalid new device string format 2024-01-19T01:21:36.830 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.152:3300" 2024-01-19T01:21:36.830 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T01:21:36.830 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: resolved to: "172.21.15.79:6789,172.21.15.152:6789" 2024-01-19T01:21:36.831 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.79:6789,172.21.15.152:6789:/ 2024-01-19T01:21:36.831 INFO:teuthology.orchestra.run.smithi079.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f3747e56-b667-11ee-95ae-87774f69a715" will pass to kernel 2024-01-19T01:21:36.831 INFO:teuthology.orchestra.run.smithi079.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-01-19T01:21:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: You just mounted a file system that supports labels which does not 2024-01-19T01:21:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-01-19T01:21:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: applications will generate AVC messages and not be allowed access to 2024-01-19T01:21:36.831 INFO:teuthology.orchestra.run.smithi079.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-01-19T01:21:36.833 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-19T01:21:36.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.833 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:36.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.861 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T01:21:36.923 INFO:teuthology.orchestra.run.smithi079.stdout:client24295 2024-01-19T01:21:36.924 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:21:36.924 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T01:21:36.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:36 smithi152 ceph-mon[35694]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2024-01-19T01:21:36.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:36.980 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T01:21:36.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:36 smithi079 ceph-mon[28374]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2024-01-19T01:21:37.020 INFO:teuthology.orchestra.run.smithi079.stdout:client24295 2024-01-19T01:21:37.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:37.020 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T01:21:37.084 INFO:teuthology.orchestra.run.smithi079.stdout:client24295 2024-01-19T01:21:37.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:37.084 DEBUG:teuthology.orchestra.run.smithi079:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f3747e56-b667-11ee-95ae-87774f69a715.client24295/status') 2024-01-19T01:21:37.155 INFO:teuthology.orchestra.run.smithi079.stdout:instance: client.24295 (3)172.21.15.79:0/304825911 2024-01-19T01:21:37.155 INFO:teuthology.orchestra.run.smithi079.stdout:blocklisted: false 2024-01-19T01:21:37.156 INFO:teuthology.orchestra.run.smithi079.stderr:0+1 records in 2024-01-19T01:21:37.156 INFO:teuthology.orchestra.run.smithi079.stderr:0+1 records out 2024-01-19T01:21:37.156 INFO:teuthology.orchestra.run.smithi079.stderr:70 bytes copied, 2.0185e-05 s, 3.5 MB/s 2024-01-19T01:21:37.157 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-01-19T01:21:37.157 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-19T01:21:37.157 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:37.157 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid f3747e56-b667-11ee-95ae-87774f69a715 -- ceph fs ls 2024-01-19T01:21:37.458 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:38.871 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:38 smithi079 ceph-mon[28374]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:38.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:38 smithi152 ceph-mon[35694]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:21:39.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 -- 172.21.15.79:0/1209251663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 msgr2=0x7f69800fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:39.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 --2- 172.21.15.79:0/1209251663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f69800fbfb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6970004940 tx=0x7f6970009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:39.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 -- 172.21.15.79:0/1209251663 shutdown_connections 2024-01-19T01:21:39.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 --2- 172.21.15.79:0/1209251663 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f69800fbfb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:39.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 --2- 172.21.15.79:0/1209251663 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6980101b00 0x7f6980101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:39.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 -- 172.21.15.79:0/1209251663 >> 172.21.15.79:0/1209251663 conn(0x7f69800719b0 msgr2=0x7f6980071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:39.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 -- 172.21.15.79:0/1209251663 shutdown_connections 2024-01-19T01:21:39.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.116+0000 7f6984a2a700 1 -- 172.21.15.79:0/1209251663 wait complete. 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f6984a2a700 1 Processor -- start 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f6984a2a700 1 -- start start 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f6984a2a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f6980112560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f6984a2a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6980101b00 0x7f6980112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f6984a2a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6980113180 con 0x7f69800fbb40 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f6984a2a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698010c420 con 0x7f6980101b00 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f697e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f6980112560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:39.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f697e59c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f6980112560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52420/0 (socket says 172.21.15.79:52420) 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f697e59c700 1 -- 172.21.15.79:0/3988994328 learned_addr learned my addr 172.21.15.79:0/3988994328 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f697dd9b700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6980101b00 0x7f6980112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f697e59c700 1 -- 172.21.15.79:0/3988994328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6980101b00 msgr2=0x7f6980112aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f697e59c700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6980101b00 0x7f6980112aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.117+0000 7f697e59c700 1 -- 172.21.15.79:0/3988994328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6970005040 con 0x7f69800fbb40 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.118+0000 7f697e59c700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f6980112560 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f697400b790 tx=0x7f6974009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.118+0000 7f696effd700 1 -- 172.21.15.79:0/3988994328 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f697400f770 con 0x7f69800fbb40 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.118+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698010c700 con 0x7f69800fbb40 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.118+0000 7f696effd700 1 -- 172.21.15.79:0/3988994328 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f697400fd80 con 0x7f69800fbb40 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.118+0000 7f696effd700 1 -- 172.21.15.79:0/3988994328 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f697400d310 con 0x7f69800fbb40 2024-01-19T01:21:39.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.118+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698010cc40 con 0x7f69800fbb40 2024-01-19T01:21:39.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.119+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6980103fb0 con 0x7f69800fbb40 2024-01-19T01:21:39.122 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.119+0000 7f696effd700 1 -- 172.21.15.79:0/3988994328 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6974010420 con 0x7f69800fbb40 2024-01-19T01:21:39.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.120+0000 7f696effd700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6968068f70 0x7f696806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:39.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.120+0000 7f696effd700 1 -- 172.21.15.79:0/3988994328 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6974086dd0 con 0x7f69800fbb40 2024-01-19T01:21:39.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.120+0000 7f697dd9b700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6968068f70 0x7f696806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:39.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.120+0000 7f697dd9b700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6968068f70 0x7f696806b420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f698010de00 tx=0x7f6970009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:39.125 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.124+0000 7f696effd700 1 -- 172.21.15.79:0/3988994328 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6974056ce0 con 0x7f69800fbb40 2024-01-19T01:21:39.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.354+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f698010db40 con 0x7f69800fbb40 2024-01-19T01:21:39.355 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.354+0000 7f696effd700 1 -- 172.21.15.79:0/3988994328 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f6974056870 con 0x7f69800fbb40 2024-01-19T01:21:39.356 INFO:teuthology.orchestra.run.smithi079.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6968068f70 msgr2=0x7f696806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6968068f70 0x7f696806b420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f698010de00 tx=0x7f6970009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 msgr2=0x7f6980112560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f6980112560 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f697400b790 tx=0x7f6974009250 comp rx=0 tx=0).stop 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 shutdown_connections 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6968068f70 0x7f696806b420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f69800fbb40 0x7f6980112560 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 --2- 172.21.15.79:0/3988994328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6980101b00 0x7f6980112aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.356+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 >> 172.21.15.79:0/3988994328 conn(0x7f69800719b0 msgr2=0x7f69800faa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:39.358 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.357+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 shutdown_connections 2024-01-19T01:21:39.359 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:39.357+0000 7f6984a2a700 1 -- 172.21.15.79:0/3988994328 wait complete. 2024-01-19T01:21:39.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:39 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3988994328' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T01:21:39.972 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:39 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3988994328' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-19T01:21:39.998 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi152.front.sepia.ceph.com 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-19T01:21:39.999 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-19T01:21:39.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:39.999 DEBUG:teuthology.orchestra.run.smithi152:> ip addr 2024-01-19T01:21:40.023 INFO:teuthology.orchestra.run.smithi152.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-19T01:21:40.023 INFO:teuthology.orchestra.run.smithi152.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-19T01:21:40.023 INFO:teuthology.orchestra.run.smithi152.stdout: inet 127.0.0.1/8 scope host lo 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft forever preferred_lft forever 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: inet6 ::1/128 scope host 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft forever preferred_lft forever 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:da:6c:44 brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: altname enp7s0f0 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:da:6c:45 brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: altname enp7s0f1 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-19T01:21:40.024 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:88:6d:e2 brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi152.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi152.stdout: link/ether 0c:c4:7a:88:6d:e3 brd ff:ff:ff:ff:ff:ff 2024-01-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi152.stdout: inet 172.21.15.152/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft 42287sec preferred_lft 42287sec 2024-01-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi152.stdout: inet6 fe80::ec4:7aff:fe88:6de3/64 scope link 2024-01-19T01:21:40.025 INFO:teuthology.orchestra.run.smithi152.stdout: valid_lft forever preferred_lft forever 2024-01-19T01:21:40.025 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-19T01:21:40.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.025 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-01-19T01:21:40.026 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-01-19T01:21:40.026 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link add name ceph-brx type bridge 2024-01-19T01:21:40.026 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip addr flush dev ceph-brx 2024-01-19T01:21:40.026 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link set ceph-brx up 2024-01-19T01:21:40.026 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-19T01:21:40.026 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-01-19T01:21:40.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.203 DEBUG:teuthology.orchestra.run.smithi152:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-19T01:21:40.231 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-01-19T01:21:40.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.233 DEBUG:teuthology.orchestra.run.smithi152:> route 2024-01-19T01:21:40.301 INFO:teuthology.orchestra.run.smithi152.stdout:Kernel IP routing table 2024-01-19T01:21:40.301 INFO:teuthology.orchestra.run.smithi152.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-19T01:21:40.301 INFO:teuthology.orchestra.run.smithi152.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-19T01:21:40.301 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-19T01:21:40.301 INFO:teuthology.orchestra.run.smithi152.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-19T01:21:40.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.302 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-01-19T01:21:40.302 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-01-19T01:21:40.302 DEBUG:teuthology.orchestra.run.smithi152:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-19T01:21:40.302 DEBUG:teuthology.orchestra.run.smithi152:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-19T01:21:40.302 DEBUG:teuthology.orchestra.run.smithi152:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-19T01:21:40.302 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-01-19T01:21:40.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.577 DEBUG:teuthology.orchestra.run.smithi152:> ip netns list 2024-01-19T01:21:40.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.592 DEBUG:teuthology.orchestra.run.smithi152:> ip netns list-id 2024-01-19T01:21:40.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.648 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-01-19T01:21:40.648 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-01-19T01:21:40.648 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-19T01:21:40.648 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-19T01:21:40.648 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-01-19T01:21:40.733 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-19T01:21:40.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.734 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-01-19T01:21:40.734 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-01-19T01:21:40.734 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-19T01:21:40.734 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-19T01:21:40.734 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-19T01:21:40.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-19T01:21:40.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-19T01:21:40.735 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-01-19T01:21:40.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:40.882 DEBUG:teuthology.orchestra.run.smithi152:> (cd / && exec bash -c ' 2024-01-19T01:21:40.882 DEBUG:teuthology.orchestra.run.smithi152:> set -e 2024-01-19T01:21:40.882 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link set brx.0 up 2024-01-19T01:21:40.882 DEBUG:teuthology.orchestra.run.smithi152:> sudo ip link set dev brx.0 master ceph-brx 2024-01-19T01:21:40.882 DEBUG:teuthology.orchestra.run.smithi152:> ') 2024-01-19T01:21:40.906 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:40 smithi152 ceph-mon[35694]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T01:21:40.925 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T01:21:40.925 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:40.980 INFO:teuthology.orchestra.run.smithi152.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-19T01:21:40.981 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-19T01:21:40.981 DEBUG:teuthology.orchestra.run.smithi152:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:40.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:40 smithi079 ceph-mon[28374]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T01:21:41.035 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-01-19T01:21:41.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:41.035 DEBUG:teuthology.orchestra.run.smithi152:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T01:21:41.140 INFO:teuthology.orchestra.run.smithi152.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T01:21:41.140 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-19T01:21:41.140 INFO:teuthology.orchestra.run.smithi152.stdout:invalid new device string format 2024-01-19T01:21:41.253 INFO:teuthology.orchestra.run.smithi152.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T01:21:41.253 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-19T01:21:41.253 INFO:teuthology.orchestra.run.smithi152.stdout:invalid new device string format 2024-01-19T01:21:41.253 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.152:3300" 2024-01-19T01:21:41.253 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:invalid new device string format 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: resolved to: "172.21.15.79:3300,172.21.15.152:3300" 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: switching to using v1 address with old syntax 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: resolved to: "172.21.15.79:6789,172.21.15.152:6789" 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.79:6789,172.21.15.152:6789:/ 2024-01-19T01:21:41.259 INFO:teuthology.orchestra.run.smithi152.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=f3747e56-b667-11ee-95ae-87774f69a715" will pass to kernel 2024-01-19T01:21:41.260 INFO:teuthology.orchestra.run.smithi152.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-01-19T01:21:41.260 INFO:teuthology.orchestra.run.smithi152.stdout: You just mounted a file system that supports labels which does not 2024-01-19T01:21:41.260 INFO:teuthology.orchestra.run.smithi152.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-01-19T01:21:41.260 INFO:teuthology.orchestra.run.smithi152.stdout: applications will generate AVC messages and not be allowed access to 2024-01-19T01:21:41.260 INFO:teuthology.orchestra.run.smithi152.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-01-19T01:21:41.261 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-01-19T01:21:41.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:41.262 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:41.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:41.289 DEBUG:teuthology.orchestra.run.smithi152:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T01:21:41.345 INFO:teuthology.orchestra.run.smithi152.stdout:client14530 2024-01-19T01:21:41.345 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:21:41.345 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-19T01:21:41.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:41.401 DEBUG:teuthology.orchestra.run.smithi152:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T01:21:41.458 INFO:teuthology.orchestra.run.smithi152.stdout:client14530 2024-01-19T01:21:41.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:41.459 DEBUG:teuthology.orchestra.run.smithi152:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-01-19T01:21:41.515 INFO:teuthology.orchestra.run.smithi152.stdout:client14530 2024-01-19T01:21:41.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-19T01:21:41.516 DEBUG:teuthology.orchestra.run.smithi152:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f3747e56-b667-11ee-95ae-87774f69a715.client14530/status') 2024-01-19T01:21:41.582 INFO:teuthology.orchestra.run.smithi152.stdout:instance: client.14530 (3)172.21.15.152:0/200066462 2024-01-19T01:21:41.582 INFO:teuthology.orchestra.run.smithi152.stdout:blocklisted: false 2024-01-19T01:21:41.584 INFO:teuthology.orchestra.run.smithi152.stderr:0+1 records in 2024-01-19T01:21:41.584 INFO:teuthology.orchestra.run.smithi152.stderr:0+1 records out 2024-01-19T01:21:41.584 INFO:teuthology.orchestra.run.smithi152.stderr:71 bytes copied, 2.2863e-05 s, 3.1 MB/s 2024-01-19T01:21:41.584 INFO:teuthology.run_tasks:Running task print... 2024-01-19T01:21:41.597 INFO:teuthology.task.print:**** done client 2024-01-19T01:21:41.597 INFO:teuthology.run_tasks:Running task parallel... 2024-01-19T01:21:41.616 INFO:teuthology.task.parallel:starting parallel... 2024-01-19T01:21:41.616 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-01-19T01:21:41.618 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-01-19T01:21:41.618 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:21:41.619 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-01-19T01:21:41.619 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-01-19T01:21:41.619 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-01-19T01:21:41.623 INFO:tasks.workunit:Pulling workunits from ref 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:41.623 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-19T01:21:41.624 INFO:tasks.workunit:timeout=3h 2024-01-19T01:21:41.624 INFO:tasks.workunit:cleanup=True 2024-01-19T01:21:41.624 DEBUG:teuthology.orchestra.run.smithi079:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:41.644 INFO:teuthology.orchestra.run.smithi079.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:41.644 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-01-19T01:21:41.644 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-01-19T01:21:41.644 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:21:41.644 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-01-19T01:21:41.644 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-01-19T01:21:41.645 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-01-19 01:21:05.606320298 +0000 2024-01-19T01:21:41.645 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-01-19 01:21:36.858914436 +0000 2024-01-19T01:21:41.645 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-01-19 01:21:05.606320298 +0000 2024-01-19T01:21:41.645 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-01-19T01:21:41.645 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-01-19T01:21:41.717 DEBUG:teuthology.orchestra.run.smithi152:> stat -- /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:41.733 INFO:teuthology.orchestra.run.smithi152.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-01-19 01:21:41.711799198 +0000 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-01-19 01:21:41.711799198 +0000 2024-01-19T01:21:41.734 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-01-19 01:21:05.606320298 +0000 2024-01-19T01:21:41.734 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-01-19T01:21:41.734 DEBUG:teuthology.orchestra.run.smithi152:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-01-19T01:21:41.805 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:41.806 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:41.822 INFO:tasks.workunit.client.0.smithi079.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-19T01:21:41.861 INFO:tasks.workunit.client.1.smithi152.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-01-19T01:21:41.900 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:42.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:42 smithi079 ceph-mon[28374]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T01:21:43.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:42 smithi152 ceph-mon[35694]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/973146389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf380fcd00 msgr2=0x7fdf380fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 --2- 172.21.15.79:0/973146389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf380fcd00 0x7fdf380fd150 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c004940 tx=0x7fdf2c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/973146389 shutdown_connections 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 --2- 172.21.15.79:0/973146389 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf380fcd00 0x7fdf380fd150 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 --2- 172.21.15.79:0/973146389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf380fbaf0 0x7fdf380fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/973146389 >> 172.21.15.79:0/973146389 conn(0x7fdf380f7190 msgr2=0x7fdf380f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/973146389 shutdown_connections 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.551+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/973146389 wait complete. 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf3d4a1700 1 Processor -- start 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf3d4a1700 1 -- start start 2024-01-19T01:21:43.554 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf3d4a1700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf380fbaf0 0x7fdf3810d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf3d4a1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf380fcd00 0x7fdf3810daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf3d4a1700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3810e130 con 0x7fdf380fbaf0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf3d4a1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3810e970 con 0x7fdf380fcd00 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf36ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf380fbaf0 0x7fdf3810d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf36ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf380fbaf0 0x7fdf3810d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52450/0 (socket says 172.21.15.79:52450) 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.552+0000 7fdf36ffd700 1 -- 172.21.15.79:0/147220221 learned_addr learned my addr 172.21.15.79:0/147220221 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf367fc700 1 --2- 172.21.15.79:0/147220221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf380fcd00 0x7fdf3810daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf36ffd700 1 -- 172.21.15.79:0/147220221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf380fcd00 msgr2=0x7fdf3810daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf36ffd700 1 --2- 172.21.15.79:0/147220221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf380fcd00 0x7fdf3810daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf36ffd700 1 -- 172.21.15.79:0/147220221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c005040 con 0x7fdf380fbaf0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf36ffd700 1 --2- 172.21.15.79:0/147220221 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdf380fbaf0 0x7fdf3810d560 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800b790 tx=0x7fdf28009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf1f7fe700 1 -- 172.21.15.79:0/147220221 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf2800f6f0 con 0x7fdf380fbaf0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf1f7fe700 1 -- 172.21.15.79:0/147220221 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf2800fd00 con 0x7fdf380fbaf0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/147220221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3810ec50 con 0x7fdf380fbaf0 2024-01-19T01:21:43.555 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf1f7fe700 1 -- 172.21.15.79:0/147220221 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf280103a0 con 0x7fdf380fbaf0 2024-01-19T01:21:43.556 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.553+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/147220221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3810f190 con 0x7fdf380fbaf0 2024-01-19T01:21:43.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.554+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/147220221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3804bb80 con 0x7fdf380fbaf0 2024-01-19T01:21:43.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.554+0000 7fdf1f7fe700 1 -- 172.21.15.79:0/147220221 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdf2800f850 con 0x7fdf380fbaf0 2024-01-19T01:21:43.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.555+0000 7fdf1f7fe700 1 --2- 172.21.15.79:0/147220221 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdf20068f60 0x7fdf2006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:43.557 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.555+0000 7fdf1f7fe700 1 -- 172.21.15.79:0/147220221 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdf28086c20 con 0x7fdf380fbaf0 2024-01-19T01:21:43.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.555+0000 7fdf367fc700 1 --2- 172.21.15.79:0/147220221 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdf20068f60 0x7fdf2006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:43.558 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.555+0000 7fdf367fc700 1 --2- 172.21.15.79:0/147220221 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdf20068f60 0x7fdf2006b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00b3b0 tx=0x7fdf2c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:43.561 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.561+0000 7fdf1f7fe700 1 -- 172.21.15.79:0/147220221 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdf28056ab0 con 0x7fdf380fbaf0 2024-01-19T01:21:43.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.746+0000 7fdf3d4a1700 1 -- 172.21.15.79:0/147220221 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7fdf38104530 con 0x7fdf380fbaf0 2024-01-19T01:21:43.747 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:43.747+0000 7fdf1f7fe700 1 -- 172.21.15.79:0/147220221 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7fdf28056640 con 0x7fdf380fbaf0 2024-01-19T01:21:43.749 INFO:teuthology.orchestra.run.smithi079.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-01-19T01:21:44.450 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-01-19T01:21:44.451 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:21:44.451 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-01-19T01:21:44.755 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:44.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:44 smithi079 ceph-mon[28374]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T01:21:45.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:44 smithi152 ceph-mon[35694]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-01-19T01:21:46.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 -- 172.21.15.79:0/2022963032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fde00 msgr2=0x7fe2580fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:46.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2022963032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fde00 0x7fe2580fe250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe24c004940 tx=0x7fe24c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:46.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 -- 172.21.15.79:0/2022963032 shutdown_connections 2024-01-19T01:21:46.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2022963032 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fde00 0x7fe2580fe250 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2022963032 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2580fcbf0 0x7fe2580fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 -- 172.21.15.79:0/2022963032 >> 172.21.15.79:0/2022963032 conn(0x7fe2580f82d0 msgr2=0x7fe2580fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 -- 172.21.15.79:0/2022963032 shutdown_connections 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 -- 172.21.15.79:0/2022963032 wait complete. 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.445+0000 7fe25c8db700 1 Processor -- start 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25c8db700 1 -- start start 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25c8db700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fcbf0 0x7fe258071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25c8db700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2580fde00 0x7fe25806ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25c8db700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe258070590 con 0x7fe2580fcbf0 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25c8db700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe258070700 con 0x7fe2580fde00 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fcbf0 0x7fe258071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25659c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fcbf0 0x7fe258071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:52478/0 (socket says 172.21.15.79:52478) 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25659c700 1 -- 172.21.15.79:0/2205354241 learned_addr learned my addr 172.21.15.79:0/2205354241 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:46.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe255d9b700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2580fde00 0x7fe25806ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25659c700 1 -- 172.21.15.79:0/2205354241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2580fde00 msgr2=0x7fe25806ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25659c700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2580fde00 0x7fe25806ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25659c700 1 -- 172.21.15.79:0/2205354241 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c005040 con 0x7fe2580fcbf0 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe25659c700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fcbf0 0x7fe258071b10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe248002a80 tx=0x7fe248004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.446+0000 7fe246ffd700 1 -- 172.21.15.79:0/2205354241 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe248007cc0 con 0x7fe2580fcbf0 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.447+0000 7fe246ffd700 1 -- 172.21.15.79:0/2205354241 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe24800f040 con 0x7fe2580fcbf0 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.447+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2580709e0 con 0x7fe2580fcbf0 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.447+0000 7fe246ffd700 1 -- 172.21.15.79:0/2205354241 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe24800da20 con 0x7fe2580fcbf0 2024-01-19T01:21:46.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.447+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe258115570 con 0x7fe2580fcbf0 2024-01-19T01:21:46.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.448+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25804bb80 con 0x7fe2580fcbf0 2024-01-19T01:21:46.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.448+0000 7fe246ffd700 1 -- 172.21.15.79:0/2205354241 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe24800db80 con 0x7fe2580fcbf0 2024-01-19T01:21:46.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.449+0000 7fe246ffd700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe240068f60 0x7fe24006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:46.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.449+0000 7fe246ffd700 1 -- 172.21.15.79:0/2205354241 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe248086bd0 con 0x7fe2580fcbf0 2024-01-19T01:21:46.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.449+0000 7fe255d9b700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe240068f60 0x7fe24006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:46.452 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.449+0000 7fe255d9b700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe240068f60 0x7fe24006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe24c004560 tx=0x7fe24c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:46.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.453+0000 7fe246ffd700 1 -- 172.21.15.79:0/2205354241 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe248056a60 con 0x7fe2580fcbf0 2024-01-19T01:21:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.649+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fe258105590 con 0x7fe2580fcbf0 2024-01-19T01:21:46.650 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.649+0000 7fe246ffd700 1 -- 172.21.15.79:0/2205354241 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fe2480565f0 con 0x7fe2580fcbf0 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe240068f60 msgr2=0x7fe24006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe240068f60 0x7fe24006b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe24c004560 tx=0x7fe24c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fcbf0 msgr2=0x7fe258071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fcbf0 0x7fe258071b10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe248002a80 tx=0x7fe248004600 comp rx=0 tx=0).stop 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 shutdown_connections 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fe240068f60 0x7fe24006b410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fe2580fcbf0 0x7fe258071b10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 --2- 172.21.15.79:0/2205354241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2580fde00 0x7fe25806ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:46.652 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 >> 172.21.15.79:0/2205354241 conn(0x7fe2580f82d0 msgr2=0x7fe258101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 shutdown_connections 2024-01-19T01:21:46.653 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:46.651+0000 7fe25c8db700 1 -- 172.21.15.79:0/2205354241 wait complete. 2024-01-19T01:21:46.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:46 smithi079 ceph-mon[28374]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-01-19T01:21:47.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:46 smithi152 ceph-mon[35694]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-01-19T01:21:47.361 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-01-19T01:21:47.641 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:48.975 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:48 smithi152 ceph-mon[35694]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-01-19T01:21:48.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:48 smithi079 ceph-mon[28374]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-01-19T01:21:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 -- 172.21.15.79:0/3484348054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 msgr2=0x7f93f00fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 --2- 172.21.15.79:0/3484348054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f00fd150 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f93e0004940 tx=0x7f93e0009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 -- 172.21.15.79:0/3484348054 shutdown_connections 2024-01-19T01:21:49.447 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 --2- 172.21.15.79:0/3484348054 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f00fd150 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 --2- 172.21.15.79:0/3484348054 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f00fbaf0 0x7f93f00fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 -- 172.21.15.79:0/3484348054 >> 172.21.15.79:0/3484348054 conn(0x7f93f00f7190 msgr2=0x7f93f00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 -- 172.21.15.79:0/3484348054 shutdown_connections 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.445+0000 7f93f7907700 1 -- 172.21.15.79:0/3484348054 wait complete. 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f7907700 1 Processor -- start 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f7907700 1 -- start start 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f7907700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f00fbaf0 0x7f93f010f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f7907700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f010fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f7907700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f01102c0 con 0x7f93f00fcd00 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f7907700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0110b60 con 0x7f93f00fbaf0 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f4ea2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f010fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f4ea2700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f010fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53356/0 (socket says 172.21.15.79:53356) 2024-01-19T01:21:49.448 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f4ea2700 1 -- 172.21.15.79:0/3062856741 learned_addr learned my addr 172.21.15.79:0/3062856741 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:49.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.446+0000 7f93f56a3700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f00fbaf0 0x7f93f010f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:49.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93f4ea2700 1 -- 172.21.15.79:0/3062856741 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f00fbaf0 msgr2=0x7f93f010f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:49.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93f4ea2700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f00fbaf0 0x7f93f010f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:49.449 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93f4ea2700 1 -- 172.21.15.79:0/3062856741 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e0005040 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93f4ea2700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f010fcc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009360 tx=0x7f93e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93e5ffb700 1 -- 172.21.15.79:0/3062856741 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e000e040 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93e5ffb700 1 -- 172.21.15.79:0/3062856741 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93e000bc50 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f0110de0 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.447+0000 7f93e5ffb700 1 -- 172.21.15.79:0/3062856741 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e001c3f0 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.448+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f0111320 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.448+0000 7f93e5ffb700 1 -- 172.21.15.79:0/3062856741 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f93e0013420 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.449+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f004bb80 con 0x7f93f00fcd00 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.449+0000 7f93e5ffb700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f93dc068f60 0x7f93dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:49.450 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.449+0000 7f93e5ffb700 1 -- 172.21.15.79:0/3062856741 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f93e0087f40 con 0x7f93f00fcd00 2024-01-19T01:21:49.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.449+0000 7f93f56a3700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f93dc068f60 0x7f93dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:49.451 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.450+0000 7f93f56a3700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f93dc068f60 0x7f93dc06b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f93f00fcb60 tx=0x7f93ec00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:49.454 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.454+0000 7f93e5ffb700 1 -- 172.21.15.79:0/3062856741 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f93e0057db0 con 0x7f93f00fcd00 2024-01-19T01:21:49.654 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.653+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f93f01044b0 con 0x7f93f00fcd00 2024-01-19T01:21:49.655 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.654+0000 7f93e5ffb700 1 -- 172.21.15.79:0/3062856741 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f93e0057940 con 0x7f93f00fcd00 2024-01-19T01:21:49.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f93dc068f60 msgr2=0x7f93dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:49.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f93dc068f60 0x7f93dc06b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f93f00fcb60 tx=0x7f93ec00b040 comp rx=0 tx=0).stop 2024-01-19T01:21:49.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 msgr2=0x7f93f010fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:49.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f010fcc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009360 tx=0x7f93e0009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:49.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 shutdown_connections 2024-01-19T01:21:49.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f93dc068f60 0x7f93dc06b410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:49.657 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f00fbaf0 0x7f93f010f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:49.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 --2- 172.21.15.79:0/3062856741 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f93f00fcd00 0x7f93f010fcc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:49.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 >> 172.21.15.79:0/3062856741 conn(0x7f93f00f7190 msgr2=0x7f93f0065370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:49.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 shutdown_connections 2024-01-19T01:21:49.658 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:49.656+0000 7f93f7907700 1 -- 172.21.15.79:0/3062856741 wait complete. 2024-01-19T01:21:49.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:50.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:21:50.343 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph config set global log_to_journald false --force' 2024-01-19T01:21:50.628 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:50.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:50 smithi079 ceph-mon[28374]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T01:21:51.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:50 smithi152 ceph-mon[35694]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-01-19T01:21:52.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 -- 172.21.15.79:0/3190124710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400fb950 msgr2=0x7f04400fe200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:52.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 --2- 172.21.15.79:0/3190124710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400fb950 0x7f04400fe200 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0434004940 tx=0x7f0434009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:52.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 -- 172.21.15.79:0/3190124710 shutdown_connections 2024-01-19T01:21:52.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 --2- 172.21.15.79:0/3190124710 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400fb950 0x7f04400fe200 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:52.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 --2- 172.21.15.79:0/3190124710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04400faf70 0x7f04400fb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:52.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 -- 172.21.15.79:0/3190124710 >> 172.21.15.79:0/3190124710 conn(0x7f04400f6950 msgr2=0x7f04400f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:52.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 -- 172.21.15.79:0/3190124710 shutdown_connections 2024-01-19T01:21:52.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 -- 172.21.15.79:0/3190124710 wait complete. 2024-01-19T01:21:52.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.458+0000 7f044698b700 1 Processor -- start 2024-01-19T01:21:52.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f044698b700 1 -- start start 2024-01-19T01:21:52.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f044698b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400faf70 0x7f044010d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:52.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f044698b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f044010dc80 0x7f044010eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f044698b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044010e140 con 0x7f04400faf70 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f044698b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044010e280 con 0x7f044010dc80 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400faf70 0x7f044010d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400faf70 0x7f044010d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53376/0 (socket says 172.21.15.79:53376) 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043ffff700 1 -- 172.21.15.79:0/2164249286 learned_addr learned my addr 172.21.15.79:0/2164249286 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043f7fe700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f044010dc80 0x7f044010eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043ffff700 1 -- 172.21.15.79:0/2164249286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f044010dc80 msgr2=0x7f044010eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043ffff700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f044010dc80 0x7f044010eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043ffff700 1 -- 172.21.15.79:0/2164249286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434005040 con 0x7f04400faf70 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.459+0000 7f043ffff700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400faf70 0x7f044010d740 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0430002a80 tx=0x7f04300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.460+0000 7f043cff9700 1 -- 172.21.15.79:0/2164249286 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0430007cc0 con 0x7f04400faf70 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.460+0000 7f043cff9700 1 -- 172.21.15.79:0/2164249286 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f043000f040 con 0x7f04400faf70 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.460+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044010f180 con 0x7f04400faf70 2024-01-19T01:21:52.463 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.460+0000 7f043cff9700 1 -- 172.21.15.79:0/2164249286 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f043000da60 con 0x7f04400faf70 2024-01-19T01:21:52.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.460+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044010f690 con 0x7f04400faf70 2024-01-19T01:21:52.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.461+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044004bb80 con 0x7f04400faf70 2024-01-19T01:21:52.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.461+0000 7f043cff9700 1 -- 172.21.15.79:0/2164249286 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f043000dbc0 con 0x7f04400faf70 2024-01-19T01:21:52.464 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.461+0000 7f043cff9700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0428068f60 0x7f042806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:52.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.462+0000 7f043cff9700 1 -- 172.21.15.79:0/2164249286 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f04300869a0 con 0x7f04400faf70 2024-01-19T01:21:52.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.462+0000 7f043f7fe700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0428068f60 0x7f042806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:52.465 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.462+0000 7f043f7fe700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0428068f60 0x7f042806b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0434004f90 tx=0x7f0434009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:52.467 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.466+0000 7f043cff9700 1 -- 172.21.15.79:0/2164249286 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f04300568b0 con 0x7f04400faf70 2024-01-19T01:21:52.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.662+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f0440065150 con 0x7f04400faf70 2024-01-19T01:21:52.663 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.662+0000 7f043cff9700 1 -- 172.21.15.79:0/2164249286 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f0430056440 con 0x7f04400faf70 2024-01-19T01:21:52.668 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0428068f60 msgr2=0x7f042806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0428068f60 0x7f042806b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0434004f90 tx=0x7f0434009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400faf70 msgr2=0x7f044010d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400faf70 0x7f044010d740 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0430002a80 tx=0x7f04300046b0 comp rx=0 tx=0).stop 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 shutdown_connections 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0428068f60 0x7f042806b410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f04400faf70 0x7f044010d740 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.664+0000 7f044698b700 1 --2- 172.21.15.79:0/2164249286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f044010dc80 0x7f044010eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.665+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 >> 172.21.15.79:0/2164249286 conn(0x7f04400f6950 msgr2=0x7f04401013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.665+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 shutdown_connections 2024-01-19T01:21:52.669 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:52.665+0000 7f044698b700 1 -- 172.21.15.79:0/2164249286 wait complete. 2024-01-19T01:21:52.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:52 smithi079 ceph-mon[28374]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:21:53.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:52 smithi152 ceph-mon[35694]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:21:53.475 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-01-19T01:21:53.744 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:54.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:54 smithi079 ceph-mon[28374]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:21:55.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:54 smithi152 ceph-mon[35694]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:21:55.602 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 -- 172.21.15.79:0/2934189038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c18066920 msgr2=0x7f0c180ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/2934189038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c18066920 0x7f0c180ff7b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c004940 tx=0x7f0c0c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 -- 172.21.15.79:0/2934189038 shutdown_connections 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/2934189038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c180ffd80 0x7f0c181021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/2934189038 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c18066920 0x7f0c180ff7b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 -- 172.21.15.79:0/2934189038 >> 172.21.15.79:0/2934189038 conn(0x7f0c180f71c0 msgr2=0x7f0c180f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 -- 172.21.15.79:0/2934189038 shutdown_connections 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 -- 172.21.15.79:0/2934189038 wait complete. 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 Processor -- start 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.600+0000 7f0c1edd9700 1 -- start start 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c1edd9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c18066920 0x7f0c1810d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c1edd9700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c180ffd80 0x7f0c1810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c1edd9700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1810e010 con 0x7f0c180ffd80 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c1edd9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1810e8b0 con 0x7f0c18066920 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c17fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c180ffd80 0x7f0c1810da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:55.603 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c17fff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c180ffd80 0x7f0c1810da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:53394/0 (socket says 172.21.15.79:53394) 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c17fff700 1 -- 172.21.15.79:0/3995715636 learned_addr learned my addr 172.21.15.79:0/3995715636 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c1cb75700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c18066920 0x7f0c1810d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c17fff700 1 -- 172.21.15.79:0/3995715636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c18066920 msgr2=0x7f0c1810d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c17fff700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c18066920 0x7f0c1810d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c17fff700 1 -- 172.21.15.79:0/3995715636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c005040 con 0x7f0c180ffd80 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c17fff700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c180ffd80 0x7f0c1810da10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400eb40 tx=0x7f0c0400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c157fa700 1 -- 172.21.15.79:0/3995715636 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c04007cc0 con 0x7f0c180ffd80 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c157fa700 1 -- 172.21.15.79:0/3995715636 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c0400f040 con 0x7f0c180ffd80 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.601+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1810eb30 con 0x7f0c180ffd80 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.602+0000 7f0c157fa700 1 -- 172.21.15.79:0/3995715636 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c0400c8f0 con 0x7f0c180ffd80 2024-01-19T01:21:55.604 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.602+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1810f070 con 0x7f0c180ffd80 2024-01-19T01:21:55.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.603+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c181075f0 con 0x7f0c180ffd80 2024-01-19T01:21:55.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.603+0000 7f0c157fa700 1 -- 172.21.15.79:0/3995715636 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0c040090d0 con 0x7f0c180ffd80 2024-01-19T01:21:55.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.603+0000 7f0c157fa700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c08068f60 0x7f0c0806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:55.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.603+0000 7f0c157fa700 1 -- 172.21.15.79:0/3995715636 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0c04086760 con 0x7f0c180ffd80 2024-01-19T01:21:55.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.603+0000 7f0c1cb75700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c08068f60 0x7f0c0806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:55.605 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.604+0000 7f0c1cb75700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c08068f60 0x7f0c0806b410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c000c00 tx=0x7f0c0c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:55.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.609+0000 7f0c157fa700 1 -- 172.21.15.79:0/3995715636 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0c0404da00 con 0x7f0c180ffd80 2024-01-19T01:21:55.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.824+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c1802b1f0 con 0x7f0c08068f60 2024-01-19T01:21:55.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.831+0000 7f0c157fa700 1 -- 172.21.15.79:0/3995715636 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f0c1802b1f0 con 0x7f0c08068f60 2024-01-19T01:21:55.832 INFO:teuthology.orchestra.run.smithi079.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:55.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c08068f60 msgr2=0x7f0c0806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:55.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c08068f60 0x7f0c0806b410 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c000c00 tx=0x7f0c0c009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:55.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c180ffd80 msgr2=0x7f0c1810da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:55.834 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c180ffd80 0x7f0c1810da10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0c0400eb40 tx=0x7f0c0400b040 comp rx=0 tx=0).stop 2024-01-19T01:21:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 shutdown_connections 2024-01-19T01:21:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0c08068f60 0x7f0c0806b410 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c18066920 0x7f0c1810d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 --2- 172.21.15.79:0/3995715636 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0c180ffd80 0x7f0c1810da10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.833+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 >> 172.21.15.79:0/3995715636 conn(0x7f0c180f71c0 msgr2=0x7f0c180f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.834+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 shutdown_connections 2024-01-19T01:21:55.835 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:55.834+0000 7f0c1edd9700 1 -- 172.21.15.79:0/3995715636 wait complete. 2024-01-19T01:21:56.605 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-01-19T01:21:56.606 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi079.front.sepia.ceph.com 2024-01-19T01:21:56.606 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f3747e56-b667-11ee-95ae-87774f69a715 -e sha1=0edf41a622739843d6b978b179ff3227b476dd9d -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-01-19T01:21:56.898 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/f3747e56-b667-11ee-95ae-87774f69a715/mon.smithi079/config 2024-01-19T01:21:56.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:56 smithi079 ceph-mon[28374]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:21:56.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:56.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:56.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:56.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:56.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:56 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:57.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:56 smithi152 ceph-mon[35694]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:21:57.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:57.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:21:57.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:21:57.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-19T01:21:57.174 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:56 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:21:57.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:57 smithi079 ceph-mon[28374]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:21:57.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:57 smithi079 ceph-mon[28374]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:57.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:57 smithi079 ceph-mon[28374]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:58.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:57 smithi152 ceph-mon[35694]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:21:58.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:57 smithi152 ceph-mon[35694]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:58.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:57 smithi152 ceph-mon[35694]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-19T01:21:58.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:21:58 smithi079 ceph-mon[28374]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 -- 172.21.15.79:0/2761635472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f80fec50 msgr2=0x7f18f80fad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 --2- 172.21.15.79:0/2761635472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f80fec50 0x7f18f80fad00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f18e8004940 tx=0x7f18e8009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 -- 172.21.15.79:0/2761635472 shutdown_connections 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 --2- 172.21.15.79:0/2761635472 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f80fec50 0x7f18f80fad00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 --2- 172.21.15.79:0/2761635472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18f80fe270 0x7f18f80fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 -- 172.21.15.79:0/2761635472 >> 172.21.15.79:0/2761635472 conn(0x7f18f80f6950 msgr2=0x7f18f80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 -- 172.21.15.79:0/2761635472 shutdown_connections 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.930+0000 7f18ff291700 1 -- 172.21.15.79:0/2761635472 wait complete. 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18ff291700 1 Processor -- start 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18ff291700 1 -- start start 2024-01-19T01:21:58.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18ff291700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18f80fe270 0x7f18f810d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18ff291700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f810dad0 0x7f18f810e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18ff291700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f810df90 con 0x7f18f810dad0 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18ff291700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f810e0d0 con 0x7f18f80fe270 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18fc82c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f810dad0 0x7f18f810e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18fc82c700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f810dad0 0x7f18f810e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36274/0 (socket says 172.21.15.79:36274) 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18fc82c700 1 -- 172.21.15.79:0/3065578749 learned_addr learned my addr 172.21.15.79:0/3065578749 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.931+0000 7f18fd02d700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18f80fe270 0x7f18f810d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18fd02d700 1 -- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f810dad0 msgr2=0x7f18f810e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18fd02d700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f810dad0 0x7f18f810e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18fd02d700 1 -- 172.21.15.79:0/3065578749 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e8005040 con 0x7f18f80fe270 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18fc82c700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f810dad0 0x7f18f810e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T01:21:58.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18fd02d700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18f80fe270 0x7f18f810d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18f8065010 tx=0x7f18f400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:58.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18edffb700 1 -- 172.21.15.79:0/3065578749 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18f4004e90 con 0x7f18f80fe270 2024-01-19T01:21:58.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18edffb700 1 -- 172.21.15.79:0/3065578749 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18f400f040 con 0x7f18f80fe270 2024-01-19T01:21:58.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f810ef70 con 0x7f18f80fe270 2024-01-19T01:21:58.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.932+0000 7f18edffb700 1 -- 172.21.15.79:0/3065578749 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18f400c890 con 0x7f18f80fe270 2024-01-19T01:21:58.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.933+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f810f480 con 0x7f18f80fe270 2024-01-19T01:21:58.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.934+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f804bb80 con 0x7f18f80fe270 2024-01-19T01:21:58.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.934+0000 7f18edffb700 1 -- 172.21.15.79:0/3065578749 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f18f400cb00 con 0x7f18f80fe270 2024-01-19T01:21:58.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.935+0000 7f18edffb700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f18e4068f70 0x7f18e406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:58.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.935+0000 7f18edffb700 1 -- 172.21.15.79:0/3065578749 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f18f4086830 con 0x7f18f80fe270 2024-01-19T01:21:58.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.935+0000 7f18fc82c700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f18e4068f70 0x7f18e406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:58.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.935+0000 7f18fc82c700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f18e4068f70 0x7f18e406b420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f18e800b0d0 tx=0x7f18e8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:58.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:58.940+0000 7f18edffb700 1 -- 172.21.15.79:0/3065578749 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f18f404da00 con 0x7f18f80fe270 2024-01-19T01:21:59.023 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:21:58 smithi152 ceph-mon[35694]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:21:59.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.265+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f18f80fecc0 con 0x7f18e4068f70 2024-01-19T01:21:59.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.266+0000 7f18edffb700 1 -- 172.21.15.79:0/3065578749 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f18f80fecc0 con 0x7f18e4068f70 2024-01-19T01:21:59.269 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f18e4068f70 msgr2=0x7f18e406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f18e4068f70 0x7f18e406b420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f18e800b0d0 tx=0x7f18e8009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18f80fe270 msgr2=0x7f18f810d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18f80fe270 0x7f18f810d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18f8065010 tx=0x7f18f400b040 comp rx=0 tx=0).stop 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 shutdown_connections 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f18e4068f70 0x7f18e406b420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18f80fe270 0x7f18f810d590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 --2- 172.21.15.79:0/3065578749 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f18f810dad0 0x7f18f810e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 >> 172.21.15.79:0/3065578749 conn(0x7f18f80f6950 msgr2=0x7f18f8101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 shutdown_connections 2024-01-19T01:21:59.270 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.268+0000 7f18ff291700 1 -- 172.21.15.79:0/3065578749 wait complete. 2024-01-19T01:21:59.283 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-01-19T01:21:59.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 -- 172.21.15.79:0/3737310173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fcbf0 msgr2=0x7f1ebc0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 --2- 172.21.15.79:0/3737310173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fcbf0 0x7f1ebc0fd000 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1eac004940 tx=0x7f1eac009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:59.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 -- 172.21.15.79:0/3737310173 shutdown_connections 2024-01-19T01:21:59.404 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 --2- 172.21.15.79:0/3737310173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ebc0fde00 0x7f1ebc0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 --2- 172.21.15.79:0/3737310173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fcbf0 0x7f1ebc0fd000 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 -- 172.21.15.79:0/3737310173 >> 172.21.15.79:0/3737310173 conn(0x7f1ebc0f82d0 msgr2=0x7f1ebc0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:59.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 -- 172.21.15.79:0/3737310173 shutdown_connections 2024-01-19T01:21:59.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 -- 172.21.15.79:0/3737310173 wait complete. 2024-01-19T01:21:59.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 Processor -- start 2024-01-19T01:21:59.405 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.403+0000 7f1ec40db700 1 -- start start 2024-01-19T01:21:59.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec40db700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ebc0fcbf0 0x7f1ebc10d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec40db700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fde00 0x7f1ebc10daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec40db700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc10e0a0 con 0x7f1ebc0fde00 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec40db700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc10e940 con 0x7f1ebc0fcbf0 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1676700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fde00 0x7f1ebc10daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1676700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fde00 0x7f1ebc10daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36296/0 (socket says 172.21.15.79:36296) 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1676700 1 -- 172.21.15.79:0/216427325 learned_addr learned my addr 172.21.15.79:0/216427325 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1e77700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ebc0fcbf0 0x7f1ebc10d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1676700 1 -- 172.21.15.79:0/216427325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ebc0fcbf0 msgr2=0x7f1ebc10d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1676700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ebc0fcbf0 0x7f1ebc10d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1676700 1 -- 172.21.15.79:0/216427325 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac005040 con 0x7f1ebc0fde00 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1ec1676700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fde00 0x7f1ebc10daa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8002a50 tx=0x7f1eb80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.404+0000 7f1eb27fc700 1 -- 172.21.15.79:0/216427325 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb8007cc0 con 0x7f1ebc0fde00 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.405+0000 7f1eb27fc700 1 -- 172.21.15.79:0/216427325 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1eb800f040 con 0x7f1ebc0fde00 2024-01-19T01:21:59.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.405+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc10ec20 con 0x7f1ebc0fde00 2024-01-19T01:21:59.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.405+0000 7f1eb27fc700 1 -- 172.21.15.79:0/216427325 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb8009540 con 0x7f1ebc0fde00 2024-01-19T01:21:59.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.405+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc10f160 con 0x7f1ebc0fde00 2024-01-19T01:21:59.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.405+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc1075d0 con 0x7f1ebc0fde00 2024-01-19T01:21:59.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.406+0000 7f1eb27fc700 1 -- 172.21.15.79:0/216427325 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1eb80096c0 con 0x7f1ebc0fde00 2024-01-19T01:21:59.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.406+0000 7f1eb27fc700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1ea8068f60 0x7f1ea806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:59.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.406+0000 7f1eb27fc700 1 -- 172.21.15.79:0/216427325 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1eb8086b10 con 0x7f1ebc0fde00 2024-01-19T01:21:59.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.406+0000 7f1ec1e77700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1ea8068f60 0x7f1ea806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:59.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.407+0000 7f1ec1e77700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1ea8068f60 0x7f1ea806b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009fd0 tx=0x7f1eac009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:59.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.411+0000 7f1eb27fc700 1 -- 172.21.15.79:0/216427325 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1eb8056a20 con 0x7f1ebc0fde00 2024-01-19T01:21:59.768 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.767+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ebc10f500 con 0x7f1ea8068f60 2024-01-19T01:21:59.769 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.768+0000 7f1eb27fc700 1 -- 172.21.15.79:0/216427325 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f1ebc10f500 con 0x7f1ea8068f60 2024-01-19T01:21:59.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1ea8068f60 msgr2=0x7f1ea806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1ea8068f60 0x7f1ea806b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009fd0 tx=0x7f1eac009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:59.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fde00 msgr2=0x7f1ebc10daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.772 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fde00 0x7f1ebc10daa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8002a50 tx=0x7f1eb80046b0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 shutdown_connections 2024-01-19T01:21:59.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1ea8068f60 0x7f1ea806b410 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009fd0 tx=0x7f1eac009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:59.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ebc0fcbf0 0x7f1ebc10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 --2- 172.21.15.79:0/216427325 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1ebc0fde00 0x7f1ebc10daa0 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8002a50 tx=0x7f1eb80046b0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.771+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 >> 172.21.15.79:0/216427325 conn(0x7f1ebc0f82d0 msgr2=0x7f1ebc0fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:59.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.772+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 shutdown_connections 2024-01-19T01:21:59.773 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.772+0000 7f1ec40db700 1 -- 172.21.15.79:0/216427325 wait complete. 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 -- 172.21.15.79:0/783109599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 msgr2=0x7fb1c80fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 --2- 172.21.15.79:0/783109599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c80fd5a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8004940 tx=0x7fb1b8009e30 comp rx=0 tx=0).stop 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 -- 172.21.15.79:0/783109599 shutdown_connections 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 --2- 172.21.15.79:0/783109599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c80fbe10 0x7fb1c80fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 --2- 172.21.15.79:0/783109599 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c80fd5a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 -- 172.21.15.79:0/783109599 >> 172.21.15.79:0/783109599 conn(0x7fb1c80f6950 msgr2=0x7fb1c80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 -- 172.21.15.79:0/783109599 shutdown_connections 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.906+0000 7fb1cd751700 1 -- 172.21.15.79:0/783109599 wait complete. 2024-01-19T01:21:59.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1cd751700 1 Processor -- start 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1cd751700 1 -- start start 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1cd751700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c80fbe10 0x7fb1c810c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1cd751700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c810c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1cd751700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c810cf90 con 0x7fb1c80fd190 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1cd751700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1c810d830 con 0x7fb1c80fbe10 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1c67fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c810c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1c67fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c810c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36314/0 (socket says 172.21.15.79:36314) 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.907+0000 7fb1c67fc700 1 -- 172.21.15.79:0/974051794 learned_addr learned my addr 172.21.15.79:0/974051794 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1c6ffd700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c80fbe10 0x7fb1c810c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1c67fc700 1 -- 172.21.15.79:0/974051794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c80fbe10 msgr2=0x7fb1c810c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1c67fc700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c80fbe10 0x7fb1c810c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1c67fc700 1 -- 172.21.15.79:0/974051794 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b8005040 con 0x7fb1c80fd190 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1c6ffd700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c80fbe10 0x7fb1c810c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T01:21:59.910 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1c67fc700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c810c990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc010d50 tx=0x7fb1bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:59.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1af7fe700 1 -- 172.21.15.79:0/974051794 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1bc00ed60 con 0x7fb1c80fd190 2024-01-19T01:21:59.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c810dab0 con 0x7fb1c80fd190 2024-01-19T01:21:59.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1af7fe700 1 -- 172.21.15.79:0/974051794 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1bc0090d0 con 0x7fb1c80fd190 2024-01-19T01:21:59.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c810dff0 con 0x7fb1c80fd190 2024-01-19T01:21:59.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.908+0000 7fb1af7fe700 1 -- 172.21.15.79:0/974051794 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1bc012650 con 0x7fb1c80fd190 2024-01-19T01:21:59.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.909+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c804bb80 con 0x7fb1c80fd190 2024-01-19T01:21:59.911 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.910+0000 7fb1af7fe700 1 -- 172.21.15.79:0/974051794 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb1bc018030 con 0x7fb1c80fd190 2024-01-19T01:21:59.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.911+0000 7fb1af7fe700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb1b0068f60 0x7fb1b006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:21:59.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.911+0000 7fb1af7fe700 1 -- 172.21.15.79:0/974051794 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb1bc016070 con 0x7fb1c80fd190 2024-01-19T01:21:59.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.911+0000 7fb1c6ffd700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb1b0068f60 0x7fb1b006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:21:59.912 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.911+0000 7fb1c6ffd700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb1b0068f60 0x7fb1b006b410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8004560 tx=0x7fb1b8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:21:59.916 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:21:59.915+0000 7fb1af7fe700 1 -- 172.21.15.79:0/974051794 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb1bc059240 con 0x7fb1c80fd190 2024-01-19T01:22:00.217 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.216+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1c810e450 con 0x7fb1b0068f60 2024-01-19T01:22:00.235 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.234+0000 7fb1af7fe700 1 -- 172.21.15.79:0/974051794 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fb1c810e450 con 0x7fb1b0068f60 2024-01-19T01:22:00.235 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-19T01:22:00.235 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.smithi079 smithi079 *:9093,9094 running (4m) 30s ago 6m 25.7M - 0.25.0 c8568f914cd2 e31a5a73bdf6 2024-01-19T01:22:00.235 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi079 smithi079 running (6m) 30s ago 6m 6665k - 17.2.7-268-g6441305d c3a232f785d8 3b02e2e6b9be 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi152 smithi152 running (5m) 36s ago 5m 6660k - 17.2.7-268-g6441305d c3a232f785d8 b47c25b14963 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.smithi079 smithi079 *:3000 running (4m) 30s ago 6m 85.2M - 9.4.7 954c08fa6188 551c78ecdb2b 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.cwngbt smithi079 running (55s) 30s ago 54s 10.4M - 17.2.7-268-g6441305d c3a232f785d8 d9755e8a61f6 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.qxcrcz smithi079 running (49s) 30s ago 48s 16.5M - 17.2.7-268-g6441305d c3a232f785d8 6bb941a74e26 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi152.btzecy smithi152 running (46s) 36s ago 46s 12.0M - 17.2.7-268-g6441305d c3a232f785d8 8bef9d65c58e 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi152.hpxjws smithi152 running (52s) 36s ago 51s 14.3M - 17.2.7-268-g6441305d c3a232f785d8 98e18f4a7c6f 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi079.miqczk smithi079 *:9283 running (7m) 30s ago 7m 479M - 17.2.7-268-g6441305d c3a232f785d8 d038b98fa75a 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi152.flyowb smithi152 *:8443,9283 running (5m) 36s ago 5m 433M - 17.2.7-268-g6441305d c3a232f785d8 92e644b493a7 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi079 smithi079 running (7m) 30s ago 7m 48.2M 2048M 17.2.7-268-g6441305d c3a232f785d8 0aba73c41ca9 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi152 smithi152 running (5m) 36s ago 5m 38.6M 2048M 17.2.7-268-g6441305d c3a232f785d8 33474ce60d80 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi079 smithi079 *:9100 running (6m) 30s ago 6m 15.9M - 1.5.0 0da6a335fe13 13236a399b50 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi152 smithi152 *:9100 running (5m) 36s ago 5m 15.4M - 1.5.0 0da6a335fe13 bc329f5a11bd 2024-01-19T01:22:00.236 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (4m) 30s ago 4m 45.6M 4288M 17.2.7-268-g6441305d c3a232f785d8 07dc267fdcd6 2024-01-19T01:22:00.237 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (4m) 30s ago 4m 45.7M 4288M 17.2.7-268-g6441305d c3a232f785d8 76e91cff255f 2024-01-19T01:22:00.237 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (3m) 30s ago 3m 45.9M 4288M 17.2.7-268-g6441305d c3a232f785d8 1fd04b4d8cd2 2024-01-19T01:22:00.237 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi152 running (3m) 36s ago 3m 43.7M 5312M 17.2.7-268-g6441305d c3a232f785d8 51d81c3e87b4 2024-01-19T01:22:00.237 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi152 running (2m) 36s ago 2m 46.3M 5312M 17.2.7-268-g6441305d c3a232f785d8 df8e00ca007c 2024-01-19T01:22:00.237 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi152 running (2m) 36s ago 2m 41.5M 5312M 17.2.7-268-g6441305d c3a232f785d8 81f6d0444b64 2024-01-19T01:22:00.237 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.smithi079 smithi079 *:9095 running (4m) 30s ago 5m 59.0M - 2.43.0 a07b618ecd1d bbe57c5d745e 2024-01-19T01:22:00.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.236+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb1b0068f60 msgr2=0x7fb1b006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.236+0000 7fb1cd751700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb1b0068f60 0x7fb1b006b410 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8004560 tx=0x7fb1b8009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:00.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.236+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 msgr2=0x7fb1c810c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.236+0000 7fb1cd751700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c810c990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc010d50 tx=0x7fb1bc00b040 comp rx=0 tx=0).stop 2024-01-19T01:22:00.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.237+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 shutdown_connections 2024-01-19T01:22:00.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.237+0000 7fb1cd751700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb1b0068f60 0x7fb1b006b410 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.238 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.237+0000 7fb1cd751700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c80fbe10 0x7fb1c810c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.237+0000 7fb1cd751700 1 --2- 172.21.15.79:0/974051794 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb1c80fd190 0x7fb1c810c990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.237+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 >> 172.21.15.79:0/974051794 conn(0x7fb1c80f6950 msgr2=0x7fb1c80f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:00.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.237+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 shutdown_connections 2024-01-19T01:22:00.239 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.237+0000 7fb1cd751700 1 -- 172.21.15.79:0/974051794 wait complete. 2024-01-19T01:22:00.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.371+0000 7f403975a700 1 -- 172.21.15.79:0/49038844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fbaf0 msgr2=0x7f40340fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.371+0000 7f403975a700 1 --2- 172.21.15.79:0/49038844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fbaf0 0x7f40340fbf00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4024004940 tx=0x7f4024009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:00.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.372+0000 7f403975a700 1 -- 172.21.15.79:0/49038844 shutdown_connections 2024-01-19T01:22:00.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.372+0000 7f403975a700 1 --2- 172.21.15.79:0/49038844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40340fcd00 0x7f40340fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.372+0000 7f403975a700 1 --2- 172.21.15.79:0/49038844 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fbaf0 0x7f40340fbf00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.372+0000 7f403975a700 1 -- 172.21.15.79:0/49038844 >> 172.21.15.79:0/49038844 conn(0x7f40340f71b0 msgr2=0x7f40340f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:00.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.372+0000 7f403975a700 1 -- 172.21.15.79:0/49038844 shutdown_connections 2024-01-19T01:22:00.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.372+0000 7f403975a700 1 -- 172.21.15.79:0/49038844 wait complete. 2024-01-19T01:22:00.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f403975a700 1 Processor -- start 2024-01-19T01:22:00.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f403975a700 1 -- start start 2024-01-19T01:22:00.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f403975a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40340fbaf0 0x7f403410d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:00.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f403975a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fcd00 0x7f403410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:00.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f403975a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403410e000 con 0x7f40340fcd00 2024-01-19T01:22:00.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f403975a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403410e8a0 con 0x7f40340fbaf0 2024-01-19T01:22:00.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f40327fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fcd00 0x7f403410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f40327fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fcd00 0x7f403410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36344/0 (socket says 172.21.15.79:36344) 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f40327fc700 1 -- 172.21.15.79:0/4267178950 learned_addr learned my addr 172.21.15.79:0/4267178950 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f40327fc700 1 -- 172.21.15.79:0/4267178950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40340fbaf0 msgr2=0x7f403410d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.373+0000 7f4032ffd700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40340fbaf0 0x7f403410d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f40327fc700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40340fbaf0 0x7f403410d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f40327fc700 1 -- 172.21.15.79:0/4267178950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4024005040 con 0x7f40340fcd00 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f4032ffd700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40340fbaf0 0x7f403410d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f40327fc700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fcd00 0x7f403410da00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f402800eb40 tx=0x7f402800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f401b7fe700 1 -- 172.21.15.79:0/4267178950 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4028007cc0 con 0x7f40340fcd00 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403410eb20 con 0x7f40340fcd00 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403410f060 con 0x7f40340fcd00 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f401b7fe700 1 -- 172.21.15.79:0/4267178950 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f402800f040 con 0x7f40340fcd00 2024-01-19T01:22:00.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.374+0000 7f401b7fe700 1 -- 172.21.15.79:0/4267178950 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4028009570 con 0x7f40340fcd00 2024-01-19T01:22:00.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.375+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40200052e0 con 0x7f40340fcd00 2024-01-19T01:22:00.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.376+0000 7f401b7fe700 1 -- 172.21.15.79:0/4267178950 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f40280096d0 con 0x7f40340fcd00 2024-01-19T01:22:00.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.376+0000 7f401b7fe700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f401c068f60 0x7f401c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:00.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.376+0000 7f401b7fe700 1 -- 172.21.15.79:0/4267178950 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f40280869b0 con 0x7f40340fcd00 2024-01-19T01:22:00.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.376+0000 7f4032ffd700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f401c068f60 0x7f401c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:00.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.377+0000 7f4032ffd700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f401c068f60 0x7f401c06b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4024004560 tx=0x7f4024009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:00.384 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.384+0000 7f401b7fe700 1 -- 172.21.15.79:0/4267178950 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f402804da00 con 0x7f40340fcd00 2024-01-19T01:22:00.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.826+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4020005da0 con 0x7f40340fcd00 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.827+0000 7f401b7fe700 1 -- 172.21.15.79:0/4267178950 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f4028056970 con 0x7f40340fcd00 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 6 2024-01-19T01:22:00.828 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:00.829 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-01-19T01:22:00.829 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 4 2024-01-19T01:22:00.829 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:00.829 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-01-19T01:22:00.829 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 14 2024-01-19T01:22:00.829 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-01-19T01:22:00.829 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f401c068f60 msgr2=0x7f401c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f401c068f60 0x7f401c06b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4024004560 tx=0x7f4024009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fcd00 msgr2=0x7f403410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fcd00 0x7f403410da00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f402800eb40 tx=0x7f402800b040 comp rx=0 tx=0).stop 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 shutdown_connections 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f401c068f60 0x7f401c06b410 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40340fbaf0 0x7f403410d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.830 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 --2- 172.21.15.79:0/4267178950 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40340fcd00 0x7f403410da00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 >> 172.21.15.79:0/4267178950 conn(0x7f40340f71b0 msgr2=0x7f40340fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:00.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 shutdown_connections 2024-01-19T01:22:00.831 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.829+0000 7f403975a700 1 -- 172.21.15.79:0/4267178950 wait complete. 2024-01-19T01:22:00.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 -- 172.21.15.79:0/1188461173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 msgr2=0x7faeb00fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 --2- 172.21.15.79:0/1188461173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 0x7faeb00fd5a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faeac004940 tx=0x7faeac009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:00.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 -- 172.21.15.79:0/1188461173 shutdown_connections 2024-01-19T01:22:00.943 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 --2- 172.21.15.79:0/1188461173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 0x7faeb00fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 --2- 172.21.15.79:0/1188461173 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 0x7faeb00fd5a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 -- 172.21.15.79:0/1188461173 >> 172.21.15.79:0/1188461173 conn(0x7faeb00f6950 msgr2=0x7faeb00f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:00.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 -- 172.21.15.79:0/1188461173 shutdown_connections 2024-01-19T01:22:00.944 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.942+0000 7faeb8030700 1 -- 172.21.15.79:0/1188461173 wait complete. 2024-01-19T01:22:00.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.943+0000 7faeb8030700 1 Processor -- start 2024-01-19T01:22:00.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.943+0000 7faeb8030700 1 -- start start 2024-01-19T01:22:00.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.943+0000 7faeb8030700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 0x7faeb010c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:00.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.943+0000 7faeb8030700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 0x7faeb010caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:00.945 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb8030700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb010d0a0 con 0x7faeb00fd190 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb8030700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb010d940 con 0x7faeb00fbe10 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb55cb700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 0x7faeb010caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb5dcc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 0x7faeb010c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb5dcc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 0x7faeb010c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.79:34200/0 (socket says 172.21.15.79:34200) 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb5dcc700 1 -- 172.21.15.79:0/2163203639 learned_addr learned my addr 172.21.15.79:0/2163203639 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb5dcc700 1 -- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 msgr2=0x7faeb010caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb5dcc700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 0x7faeb010caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb5dcc700 1 -- 172.21.15.79:0/2163203639 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac005040 con 0x7faeb00fbe10 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb55cb700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 0x7faeb010caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faeb5dcc700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 0x7faeb010c560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faeac009150 tx=0x7faeac009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.944+0000 7faea67fc700 1 -- 172.21.15.79:0/2163203639 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeac00e040 con 0x7faeb00fbe10 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.945+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb010dbc0 con 0x7faeb00fbe10 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.945+0000 7faea67fc700 1 -- 172.21.15.79:0/2163203639 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faeac009af0 con 0x7faeb00fbe10 2024-01-19T01:22:00.946 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.945+0000 7faea67fc700 1 -- 172.21.15.79:0/2163203639 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeac01c3f0 con 0x7faeb00fbe10 2024-01-19T01:22:00.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.945+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb010e100 con 0x7faeb00fbe10 2024-01-19T01:22:00.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.946+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb004bb80 con 0x7faeb00fbe10 2024-01-19T01:22:00.947 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.946+0000 7faea67fc700 1 -- 172.21.15.79:0/2163203639 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7faeac013770 con 0x7faeb00fbe10 2024-01-19T01:22:00.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.947+0000 7faea67fc700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fae9c069030 0x7fae9c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:00.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.947+0000 7faea67fc700 1 -- 172.21.15.79:0/2163203639 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7faeac088060 con 0x7faeb00fbe10 2024-01-19T01:22:00.948 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.947+0000 7faeb55cb700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fae9c069030 0x7fae9c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:00.949 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.948+0000 7faeb55cb700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fae9c069030 0x7fae9c06b4e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faea00041d0 tx=0x7faea0004470 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:00.952 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:00.951+0000 7faea67fc700 1 -- 172.21.15.79:0/2163203639 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7faeac057f50 con 0x7faeb00fbe10 2024-01-19T01:22:00.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:00 smithi079 ceph-mon[28374]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:22:00.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:00 smithi079 ceph-mon[28374]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:00.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:00 smithi079 ceph-mon[28374]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:01.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:00 smithi152 ceph-mon[35694]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:22:01.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:00 smithi152 ceph-mon[35694]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:01.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:00 smithi152 ceph-mon[35694]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:01.258 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.258+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faeb010dd50 con 0x7faeb00fbe10 2024-01-19T01:22:01.259 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.258+0000 7faea67fc700 1 -- 172.21.15.79:0/2163203639 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1809 (secure 0 0 0) 0x7faeb010dd50 con 0x7faeb00fbe10 2024-01-19T01:22:01.259 INFO:teuthology.orchestra.run.smithi079.stdout:e15 2024-01-19T01:22:01.259 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T01:22:01.259 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:22:01.259 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-01-19T01:22:01.259 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 15 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-01-19T01:21:01.585027+0000 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-01-19T01:21:18.569403+0000 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 60 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-01-19T01:22:01.260 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 46 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 1 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:in 0 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:up {0=14472} 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data enabled 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 1 2024-01-19T01:22:01.261 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.qxcrcz{0:14472} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stdout:Standby daemons: 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.cwngbt{-1:14484} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.79:6826/2748689961,v1:172.21.15.79:6827/2748689961] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.hpxjws{-1:24255} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.btzecy{-1:24265} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fae9c069030 msgr2=0x7fae9c06b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fae9c069030 0x7fae9c06b4e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faea00041d0 tx=0x7faea0004470 comp rx=0 tx=0).stop 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 msgr2=0x7faeb010c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 0x7faeb010c560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faeac009150 tx=0x7faeac009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 shutdown_connections 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fae9c069030 0x7fae9c06b4e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faeb00fbe10 0x7faeb010c560 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 --2- 172.21.15.79:0/2163203639 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7faeb00fd190 0x7faeb010caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 >> 172.21.15.79:0/2163203639 conn(0x7faeb00f6950 msgr2=0x7faeb00f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 shutdown_connections 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.261+0000 7faeb8030700 1 -- 172.21.15.79:0/2163203639 wait complete. 2024-01-19T01:22:01.263 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 15 2024-01-19T01:22:01.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 -- 172.21.15.79:0/733143748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 msgr2=0x7fdfec0fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 --2- 172.21.15.79:0/733143748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 0x7fdfec0fd150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0004940 tx=0x7fdfe0009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:01.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 -- 172.21.15.79:0/733143748 shutdown_connections 2024-01-19T01:22:01.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 --2- 172.21.15.79:0/733143748 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 0x7fdfec0fd150 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 --2- 172.21.15.79:0/733143748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 0x7fdfec0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 -- 172.21.15.79:0/733143748 >> 172.21.15.79:0/733143748 conn(0x7fdfec0f71b0 msgr2=0x7fdfec0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:01.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 -- 172.21.15.79:0/733143748 shutdown_connections 2024-01-19T01:22:01.412 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.410+0000 7fdff2644700 1 -- 172.21.15.79:0/733143748 wait complete. 2024-01-19T01:22:01.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.411+0000 7fdff2644700 1 Processor -- start 2024-01-19T01:22:01.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.411+0000 7fdff2644700 1 -- start start 2024-01-19T01:22:01.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.411+0000 7fdff2644700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 0x7fdfec113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:01.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.411+0000 7fdff2644700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 0x7fdfec113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:01.413 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdff2644700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec1145c0 con 0x7fdfec0fcd00 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdff2644700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec114e60 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfebfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 0x7fdfec113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfeb7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 0x7fdfec113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfebfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 0x7fdfec113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.79:34220/0 (socket says 172.21.15.79:34220) 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfebfff700 1 -- 172.21.15.79:0/2886947610 learned_addr learned my addr 172.21.15.79:0/2886947610 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfebfff700 1 -- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 msgr2=0x7fdfec113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfebfff700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 0x7fdfec113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfebfff700 1 -- 172.21.15.79:0/2886947610 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc005040 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.412+0000 7fdfeb7fe700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 0x7fdfec113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T01:22:01.414 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.413+0000 7fdfebfff700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 0x7fdfec113a80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0096d0 tx=0x7fdfdc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:01.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.413+0000 7fdfe8ff9700 1 -- 172.21.15.79:0/2886947610 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfdc00e7e0 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.413+0000 7fdfe8ff9700 1 -- 172.21.15.79:0/2886947610 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfdc00f040 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.413+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe0005040 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.413+0000 7fdfe8ff9700 1 -- 172.21.15.79:0/2886947610 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfdc013c90 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.415 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.413+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec115490 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.414+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec04bb80 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.414+0000 7fdfe8ff9700 1 -- 172.21.15.79:0/2886947610 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdfdc00e940 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.415+0000 7fdfe8ff9700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdfd4068f60 0x7fdfd406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:01.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.415+0000 7fdfe8ff9700 1 -- 172.21.15.79:0/2886947610 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdfdc086be0 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.416 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.415+0000 7fdfeb7fe700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdfd4068f60 0x7fdfd406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:01.417 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.415+0000 7fdfeb7fe700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdfd4068f60 0x7fdfd406b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0000c00 tx=0x7fdfe0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:01.420 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.419+0000 7fdfe8ff9700 1 -- 172.21.15.79:0/2886947610 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdfdc056a70 con 0x7fdfec0fbaf0 2024-01-19T01:22:01.688 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.687+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfec115b80 con 0x7fdfd4068f60 2024-01-19T01:22:01.689 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.689+0000 7fdfe8ff9700 1 -- 172.21.15.79:0/2886947610 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fdfec115b80 con 0x7fdfd4068f60 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d", 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout: "in_progress": true, 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout: "which": "Upgrading all daemon types on all hosts", 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout: "services_complete": [], 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout: "progress": "", 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:0edf41a622739843d6b978b179ff3227b476dd9d image", 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout: "is_paused": false 2024-01-19T01:22:01.690 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-01-19T01:22:01.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdfd4068f60 msgr2=0x7fdfd406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdfd4068f60 0x7fdfd406b410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0000c00 tx=0x7fdfe0009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:01.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 msgr2=0x7fdfec113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.692 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 0x7fdfec113a80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0096d0 tx=0x7fdfdc009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:01.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 shutdown_connections 2024-01-19T01:22:01.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fdfd4068f60 0x7fdfd406b410 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfec0fbaf0 0x7fdfec113a80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 --2- 172.21.15.79:0/2886947610 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fdfec0fcd00 0x7fdfec113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 >> 172.21.15.79:0/2886947610 conn(0x7fdfec0f71b0 msgr2=0x7fdfec0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:01.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.691+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 shutdown_connections 2024-01-19T01:22:01.693 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.692+0000 7fdff2644700 1 -- 172.21.15.79:0/2886947610 wait complete. 2024-01-19T01:22:01.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 -- 172.21.15.79:0/3405505900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 msgr2=0x7f0fd00fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/3405505900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd00fd150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4004940 tx=0x7f0fc4009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:01.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 -- 172.21.15.79:0/3405505900 shutdown_connections 2024-01-19T01:22:01.822 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/3405505900 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd00fd150 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/3405505900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fd00fbaf0 0x7f0fd00fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 -- 172.21.15.79:0/3405505900 >> 172.21.15.79:0/3405505900 conn(0x7f0fd00f71d0 msgr2=0x7f0fd00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:01.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 -- 172.21.15.79:0/3405505900 shutdown_connections 2024-01-19T01:22:01.823 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.821+0000 7f0fd65d7700 1 -- 172.21.15.79:0/3405505900 wait complete. 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.822+0000 7f0fd65d7700 1 Processor -- start 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.822+0000 7f0fd65d7700 1 -- start start 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.822+0000 7f0fd65d7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fd00fbaf0 0x7f0fd00662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.822+0000 7f0fd65d7700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd006e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.822+0000 7f0fd65d7700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0066960 con 0x7f0fd00fcd00 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.822+0000 7f0fd65d7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0066ad0 con 0x7f0fd00fbaf0 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcf7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd006e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fd00fbaf0 0x7f0fd00662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcf7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd006e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:36380/0 (socket says 172.21.15.79:36380) 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcf7fe700 1 -- 172.21.15.79:0/2685499994 learned_addr learned my addr 172.21.15.79:0/2685499994 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcf7fe700 1 -- 172.21.15.79:0/2685499994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fd00fbaf0 msgr2=0x7f0fd00662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:01.824 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcf7fe700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fd00fbaf0 0x7f0fd00662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcf7fe700 1 -- 172.21.15.79:0/2685499994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4005040 con 0x7f0fd00fcd00 2024-01-19T01:22:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcffff700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fd00fbaf0 0x7f0fd00662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-19T01:22:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fcf7fe700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd006e830 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4000c00 tx=0x7f0fc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fccff9700 1 -- 172.21.15.79:0/2685499994 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fc4021070 con 0x7f0fd00fcd00 2024-01-19T01:22:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd006ed70 con 0x7f0fd00fcd00 2024-01-19T01:22:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fccff9700 1 -- 172.21.15.79:0/2685499994 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fc401c400 con 0x7f0fd00fcd00 2024-01-19T01:22:01.825 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fccff9700 1 -- 172.21.15.79:0/2685499994 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fc400e070 con 0x7f0fd00fcd00 2024-01-19T01:22:01.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.823+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd006f2b0 con 0x7f0fd00fcd00 2024-01-19T01:22:01.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.825+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd004bb80 con 0x7f0fd00fcd00 2024-01-19T01:22:01.826 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.825+0000 7f0fccff9700 1 -- 172.21.15.79:0/2685499994 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0fc401c9c0 con 0x7f0fd00fcd00 2024-01-19T01:22:01.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.826+0000 7f0fccff9700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0fb8068f60 0x7f0fb806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:01.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.826+0000 7f0fccff9700 1 -- 172.21.15.79:0/2685499994 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0fc4087f00 con 0x7f0fd00fcd00 2024-01-19T01:22:01.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.826+0000 7f0fcffff700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0fb8068f60 0x7f0fb806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:01.827 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.827+0000 7f0fcffff700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0fb8068f60 0x7f0fb806b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0fd006f920 tx=0x7f0fc000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:01.832 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:01.831+0000 7f0fccff9700 1 -- 172.21.15.79:0/2685499994 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0fc4054420 con 0x7f0fd00fcd00 2024-01-19T01:22:01.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:01 smithi079 ceph-mon[28374]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:01.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:01 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/4267178950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T01:22:01.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:01 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2163203639' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:22:02.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:01 smithi152 ceph-mon[35694]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:02.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:01 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/4267178950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T01:22:02.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:01 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2163203639' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:22:02.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.218+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0fd01044b0 con 0x7f0fd00fcd00 2024-01-19T01:22:02.219 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.219+0000 7f0fccff9700 1 -- 172.21.15.79:0/2685499994 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f0fc4057a40 con 0x7f0fd00fcd00 2024-01-19T01:22:02.220 INFO:teuthology.orchestra.run.smithi079.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-01-19T01:22:02.220 INFO:teuthology.orchestra.run.smithi079.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-01-19T01:22:02.220 INFO:teuthology.orchestra.run.smithi079.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-01-19T01:22:02.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0fb8068f60 msgr2=0x7f0fb806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:02.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0fb8068f60 0x7f0fb806b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0fd006f920 tx=0x7f0fc000b040 comp rx=0 tx=0).stop 2024-01-19T01:22:02.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 msgr2=0x7f0fd006e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:02.222 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd006e830 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4000c00 tx=0x7f0fc4009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:02.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 shutdown_connections 2024-01-19T01:22:02.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0fb8068f60 0x7f0fb806b410 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:02.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fd00fbaf0 0x7f0fd00662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:02.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 --2- 172.21.15.79:0/2685499994 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0fd00fcd00 0x7f0fd006e830 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:02.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.221+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 >> 172.21.15.79:0/2685499994 conn(0x7f0fd00f71d0 msgr2=0x7f0fd00fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:02.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.222+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 shutdown_connections 2024-01-19T01:22:02.223 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:02.222+0000 7f0fd65d7700 1 -- 172.21.15.79:0/2685499994 wait complete. 2024-01-19T01:22:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:02 smithi079 ceph-mon[28374]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:02 smithi079 ceph-mon[28374]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:02 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2685499994' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-01-19T01:22:03.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:02 smithi152 ceph-mon[35694]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:03.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:02 smithi152 ceph-mon[35694]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:03.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:02 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2685499994' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-01-19T01:22:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:04 smithi079 ceph-mon[28374]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:05.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:04 smithi152 ceph-mon[35694]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:05.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:06.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:06 smithi079 ceph-mon[28374]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:07.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:06 smithi152 ceph-mon[35694]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:08.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:08 smithi079 ceph-mon[28374]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:09.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:08 smithi152 ceph-mon[35694]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:10 smithi079 ceph-mon[28374]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:11.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:10 smithi152 ceph-mon[35694]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:12 smithi079 ceph-mon[28374]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:13.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:12 smithi152 ceph-mon[35694]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:14.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:14 smithi079 ceph-mon[28374]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:15.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:14 smithi152 ceph-mon[35694]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:16.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:16 smithi079 ceph-mon[28374]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:17.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:16 smithi152 ceph-mon[35694]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:18.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:18 smithi079 ceph-mon[28374]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:19.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:18 smithi152 ceph-mon[35694]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:19.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:19 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:20.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:19 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:20.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:20 smithi079 ceph-mon[28374]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T01:22:21.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:20 smithi152 ceph-mon[35694]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T01:22:22.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:22 smithi079 ceph-mon[28374]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T01:22:23.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:22 smithi152 ceph-mon[35694]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-01-19T01:22:24.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:24 smithi079 ceph-mon[28374]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:25.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:24 smithi152 ceph-mon[35694]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:26.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:26 smithi079 ceph-mon[28374]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:27.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:26 smithi152 ceph-mon[35694]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:28.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:28 smithi079 ceph-mon[28374]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:28.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:28 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:22:28.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:28 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:22:28.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:28 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:22:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:28 smithi152 ceph-mon[35694]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:28 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:22:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:28 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:22:29.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:28 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:22:29.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:29 smithi079 ceph-mon[28374]: Upgrade: Target is version 19.0.0-766-g0edf41a6 (unknown) 2024-01-19T01:22:29.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:29 smithi079 ceph-mon[28374]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7'] 2024-01-19T01:22:29.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:29 smithi079 ceph-mon[28374]: Upgrade: Need to upgrade myself (mgr.smithi079.miqczk) 2024-01-19T01:22:29.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:29 smithi079 ceph-mon[28374]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 on smithi152 2024-01-19T01:22:30.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:29 smithi152 ceph-mon[35694]: Upgrade: Target is version 19.0.0-766-g0edf41a6 (unknown) 2024-01-19T01:22:30.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:29 smithi152 ceph-mon[35694]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7'] 2024-01-19T01:22:30.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:29 smithi152 ceph-mon[35694]: Upgrade: Need to upgrade myself (mgr.smithi079.miqczk) 2024-01-19T01:22:30.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:29 smithi152 ceph-mon[35694]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 on smithi152 2024-01-19T01:22:30.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:30 smithi079 ceph-mon[28374]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:31.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:30 smithi152 ceph-mon[35694]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-01-19T01:22:32.372 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 -- 172.21.15.79:0/3559439582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a40fd730 msgr2=0x7f39a40ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:32.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 --2- 172.21.15.79:0/3559439582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a40fd730 0x7f39a40ffb10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3994004940 tx=0x7f3994009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:32.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 -- 172.21.15.79:0/3559439582 shutdown_connections 2024-01-19T01:22:32.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 --2- 172.21.15.79:0/3559439582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39a4100050 0x7f39a4102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 --2- 172.21.15.79:0/3559439582 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a40fd730 0x7f39a40ffb10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 -- 172.21.15.79:0/3559439582 >> 172.21.15.79:0/3559439582 conn(0x7f39a40f7190 msgr2=0x7f39a40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:32.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 -- 172.21.15.79:0/3559439582 shutdown_connections 2024-01-19T01:22:32.373 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.371+0000 7f39a957b700 1 -- 172.21.15.79:0/3559439582 wait complete. 2024-01-19T01:22:32.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.372+0000 7f39a957b700 1 Processor -- start 2024-01-19T01:22:32.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.372+0000 7f39a957b700 1 -- start start 2024-01-19T01:22:32.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a957b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39a40fd730 0x7f39a410d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:32.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a957b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a4100050 0x7f39a410d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:32.375 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a957b700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a410dfd0 con 0x7f39a4100050 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a957b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a410e870 con 0x7f39a40fd730 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a27fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a4100050 0x7f39a410d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a27fc700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a4100050 0x7f39a410d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55448/0 (socket says 172.21.15.79:55448) 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a27fc700 1 -- 172.21.15.79:0/1411395960 learned_addr learned my addr 172.21.15.79:0/1411395960 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a2ffd700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39a40fd730 0x7f39a410d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a27fc700 1 -- 172.21.15.79:0/1411395960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39a40fd730 msgr2=0x7f39a410d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a27fc700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39a40fd730 0x7f39a410d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a27fc700 1 -- 172.21.15.79:0/1411395960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994005040 con 0x7f39a4100050 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.373+0000 7f39a27fc700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a4100050 0x7f39a410d9d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f399800eb40 tx=0x7f399800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.374+0000 7f398b7fe700 1 -- 172.21.15.79:0/1411395960 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3998007cc0 con 0x7f39a4100050 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.374+0000 7f398b7fe700 1 -- 172.21.15.79:0/1411395960 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f399800f040 con 0x7f39a4100050 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.374+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a410eaf0 con 0x7f39a4100050 2024-01-19T01:22:32.376 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.374+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a410f030 con 0x7f39a4100050 2024-01-19T01:22:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.374+0000 7f398b7fe700 1 -- 172.21.15.79:0/1411395960 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f399800c990 con 0x7f39a4100050 2024-01-19T01:22:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.375+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a404bb80 con 0x7f39a4100050 2024-01-19T01:22:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.376+0000 7f398b7fe700 1 -- 172.21.15.79:0/1411395960 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f399800caf0 con 0x7f39a4100050 2024-01-19T01:22:32.377 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.376+0000 7f398b7fe700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f398c068f70 0x7f398c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:32.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.376+0000 7f398b7fe700 1 -- 172.21.15.79:0/1411395960 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f39980869f0 con 0x7f39a4100050 2024-01-19T01:22:32.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.376+0000 7f39a2ffd700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f398c068f70 0x7f398c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:32.378 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.377+0000 7f39a2ffd700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f398c068f70 0x7f398c06b420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3994013680 tx=0x7f3994009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:32.385 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.384+0000 7f398b7fe700 1 -- 172.21.15.79:0/1411395960 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f399804da00 con 0x7f39a4100050 2024-01-19T01:22:32.673 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.672+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39a410f4d0 con 0x7f398c068f70 2024-01-19T01:22:32.674 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.674+0000 7f398b7fe700 1 -- 172.21.15.79:0/1411395960 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f39a410f4d0 con 0x7f398c068f70 2024-01-19T01:22:32.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.676+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f398c068f70 msgr2=0x7f398c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:32.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.676+0000 7f39a957b700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f398c068f70 0x7f398c06b420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3994013680 tx=0x7f3994009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:32.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.676+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a4100050 msgr2=0x7f39a410d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:32.677 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.676+0000 7f39a957b700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a4100050 0x7f39a410d9d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f399800eb40 tx=0x7f399800b040 comp rx=0 tx=0).stop 2024-01-19T01:22:32.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.677+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 shutdown_connections 2024-01-19T01:22:32.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.677+0000 7f39a957b700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f398c068f70 0x7f398c06b420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.677+0000 7f39a957b700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f39a40fd730 0x7f39a410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.677+0000 7f39a957b700 1 --2- 172.21.15.79:0/1411395960 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f39a4100050 0x7f39a410d9d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.677+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 >> 172.21.15.79:0/1411395960 conn(0x7f39a40f7190 msgr2=0x7f39a40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:32.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.677+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 shutdown_connections 2024-01-19T01:22:32.678 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.677+0000 7f39a957b700 1 -- 172.21.15.79:0/1411395960 wait complete. 2024-01-19T01:22:32.688 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-01-19T01:22:32.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 -- 172.21.15.79:0/3121993464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 msgr2=0x7f447c0ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:32.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 --2- 172.21.15.79:0/3121993464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0ffb10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f446c004940 tx=0x7f446c009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:32.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 -- 172.21.15.79:0/3121993464 shutdown_connections 2024-01-19T01:22:32.797 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 --2- 172.21.15.79:0/3121993464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f447c100050 0x7f447c102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 --2- 172.21.15.79:0/3121993464 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0ffb10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 -- 172.21.15.79:0/3121993464 >> 172.21.15.79:0/3121993464 conn(0x7f447c0f71d0 msgr2=0x7f447c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:32.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 -- 172.21.15.79:0/3121993464 shutdown_connections 2024-01-19T01:22:32.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.796+0000 7f4482141700 1 -- 172.21.15.79:0/3121993464 wait complete. 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.797+0000 7f4482141700 1 Processor -- start 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.797+0000 7f4482141700 1 -- start start 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.797+0000 7f4482141700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.797+0000 7f4482141700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f447c100050 0x7f447c0fb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.797+0000 7f4482141700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c111140 con 0x7f447c0fd730 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.797+0000 7f4482141700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c1112b0 con 0x7f447c100050 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447affd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f447c100050 0x7f447c0fb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:32.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447b7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55470/0 (socket says 172.21.15.79:55470) 2024-01-19T01:22:32.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447b7fe700 1 -- 172.21.15.79:0/358249050 learned_addr learned my addr 172.21.15.79:0/358249050 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:32.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447b7fe700 1 -- 172.21.15.79:0/358249050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f447c100050 msgr2=0x7f447c0fb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:32.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447b7fe700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f447c100050 0x7f447c0fb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:32.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447b7fe700 1 -- 172.21.15.79:0/358249050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c005040 con 0x7f447c0fd730 2024-01-19T01:22:32.800 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f447b7fe700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0fd1c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f446c000c00 tx=0x7f446c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:32.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f4463fff700 1 -- 172.21.15.79:0/358249050 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446c021070 con 0x7f447c0fd730 2024-01-19T01:22:32.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.798+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c0fbc40 con 0x7f447c0fd730 2024-01-19T01:22:32.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.799+0000 7f4463fff700 1 -- 172.21.15.79:0/358249050 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f446c01c400 con 0x7f447c0fd730 2024-01-19T01:22:32.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.799+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c0fc120 con 0x7f447c0fd730 2024-01-19T01:22:32.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.799+0000 7f4463fff700 1 -- 172.21.15.79:0/358249050 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f446c00e070 con 0x7f447c0fd730 2024-01-19T01:22:32.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.800+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44680052e0 con 0x7f447c0fd730 2024-01-19T01:22:32.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.800+0000 7f4463fff700 1 -- 172.21.15.79:0/358249050 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f446c013a30 con 0x7f447c0fd730 2024-01-19T01:22:32.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.801+0000 7f4463fff700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4464068f60 0x7f446406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:32.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.801+0000 7f4463fff700 1 -- 172.21.15.79:0/358249050 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f446c087f50 con 0x7f447c0fd730 2024-01-19T01:22:32.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.801+0000 7f447affd700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4464068f60 0x7f446406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:32.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.802+0000 7f447affd700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4464068f60 0x7f446406b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4470009b80 tx=0x7f447000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:32.809 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:32.809+0000 7f4463fff700 1 -- 172.21.15.79:0/358249050 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f446c0b9a00 con 0x7f447c0fd730 2024-01-19T01:22:32.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:32 smithi079 ceph-mon[28374]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:22:33.087 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.086+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4468000bf0 con 0x7f4464068f60 2024-01-19T01:22:33.088 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.087+0000 7f4463fff700 1 -- 172.21.15.79:0/358249050 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f4468000bf0 con 0x7f4464068f60 2024-01-19T01:22:33.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4464068f60 msgr2=0x7f446406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4464068f60 0x7f446406b410 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4470009b80 tx=0x7f447000b040 comp rx=0 tx=0).stop 2024-01-19T01:22:33.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 msgr2=0x7f447c0fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.091 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0fd1c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f446c000c00 tx=0x7f446c009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:33.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 shutdown_connections 2024-01-19T01:22:33.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4464068f60 0x7f446406b410 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f447c0fd730 0x7f447c0fd1c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 --2- 172.21.15.79:0/358249050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f447c100050 0x7f447c0fb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.090+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 >> 172.21.15.79:0/358249050 conn(0x7f447c0f71d0 msgr2=0x7f447c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:33.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.091+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 shutdown_connections 2024-01-19T01:22:33.092 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.091+0000 7f4482141700 1 -- 172.21.15.79:0/358249050 wait complete. 2024-01-19T01:22:33.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:32 smithi152 ceph-mon[35694]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.246+0000 7fb876353700 1 -- 172.21.15.79:0/391904009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fbaf0 msgr2=0x7fb8700fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.247+0000 7fb876353700 1 --2- 172.21.15.79:0/391904009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fbaf0 0x7fb8700fbf00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb860004940 tx=0x7fb860009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.247+0000 7fb876353700 1 -- 172.21.15.79:0/391904009 shutdown_connections 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.247+0000 7fb876353700 1 --2- 172.21.15.79:0/391904009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fcd00 0x7fb8700fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.247+0000 7fb876353700 1 --2- 172.21.15.79:0/391904009 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fbaf0 0x7fb8700fbf00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.247+0000 7fb876353700 1 -- 172.21.15.79:0/391904009 >> 172.21.15.79:0/391904009 conn(0x7fb8700f7190 msgr2=0x7fb8700f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.247+0000 7fb876353700 1 -- 172.21.15.79:0/391904009 shutdown_connections 2024-01-19T01:22:33.248 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.247+0000 7fb876353700 1 -- 172.21.15.79:0/391904009 wait complete. 2024-01-19T01:22:33.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb876353700 1 Processor -- start 2024-01-19T01:22:33.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb876353700 1 -- start start 2024-01-19T01:22:33.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb876353700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fbaf0 0x7fb87010c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:33.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb876353700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fcd00 0x7fb87010c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:33.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb876353700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87010cf30 con 0x7fb8700fcd00 2024-01-19T01:22:33.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb876353700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87010d7d0 con 0x7fb8700fbaf0 2024-01-19T01:22:33.250 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb86f7fe700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fcd00 0x7fb87010c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb86ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fbaf0 0x7fb87010c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb86ffff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fbaf0 0x7fb87010c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.79:42056/0 (socket says 172.21.15.79:42056) 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.248+0000 7fb86ffff700 1 -- 172.21.15.79:0/455145405 learned_addr learned my addr 172.21.15.79:0/455145405 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb86ffff700 1 -- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fcd00 msgr2=0x7fb87010c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb86ffff700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fcd00 0x7fb87010c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb86ffff700 1 -- 172.21.15.79:0/455145405 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb860005040 con 0x7fb8700fbaf0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb86f7fe700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fcd00 0x7fb87010c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb86ffff700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fbaf0 0x7fb87010c3f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb860004890 tx=0x7fb860009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb86cff9700 1 -- 172.21.15.79:0/455145405 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb86000e030 con 0x7fb8700fbaf0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87010da50 con 0x7fb8700fbaf0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.249+0000 7fb86cff9700 1 -- 172.21.15.79:0/455145405 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb860013c40 con 0x7fb8700fbaf0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.250+0000 7fb86cff9700 1 -- 172.21.15.79:0/455145405 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb860023630 con 0x7fb8700fbaf0 2024-01-19T01:22:33.251 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.250+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87010df90 con 0x7fb8700fbaf0 2024-01-19T01:22:33.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.251+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87004bb80 con 0x7fb8700fbaf0 2024-01-19T01:22:33.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.251+0000 7fb86cff9700 1 -- 172.21.15.79:0/455145405 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb86001aaa0 con 0x7fb8700fbaf0 2024-01-19T01:22:33.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.251+0000 7fb86cff9700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb858069030 0x7fb85806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:33.252 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.252+0000 7fb86cff9700 1 -- 172.21.15.79:0/455145405 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb86008a780 con 0x7fb8700fbaf0 2024-01-19T01:22:33.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.252+0000 7fb86f7fe700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb858069030 0x7fb85806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:33.253 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.252+0000 7fb86f7fe700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb858069030 0x7fb85806b4e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb864004600 tx=0x7fb86400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:33.260 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.259+0000 7fb86cff9700 1 -- 172.21.15.79:0/455145405 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb86005a670 con 0x7fb8700fbaf0 2024-01-19T01:22:33.482 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.481+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb87010e3f0 con 0x7fb858069030 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.492+0000 7fb86cff9700 1 -- 172.21.15.79:0/455145405 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fb87010e3f0 con 0x7fb858069030 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.smithi079 smithi079 *:9093,9094 running (5m) 63s ago 6m 25.7M - 0.25.0 c8568f914cd2 e31a5a73bdf6 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi079 smithi079 running (7m) 63s ago 7m 6665k - 17.2.7-268-g6441305d c3a232f785d8 3b02e2e6b9be 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi152 smithi152 running (6m) 69s ago 6m 6660k - 17.2.7-268-g6441305d c3a232f785d8 b47c25b14963 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.smithi079 smithi079 *:3000 running (5m) 63s ago 6m 85.2M - 9.4.7 954c08fa6188 551c78ecdb2b 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.cwngbt smithi079 running (88s) 63s ago 87s 10.4M - 17.2.7-268-g6441305d c3a232f785d8 d9755e8a61f6 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.qxcrcz smithi079 running (82s) 63s ago 82s 16.5M - 17.2.7-268-g6441305d c3a232f785d8 6bb941a74e26 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi152.btzecy smithi152 running (80s) 69s ago 79s 12.0M - 17.2.7-268-g6441305d c3a232f785d8 8bef9d65c58e 2024-01-19T01:22:33.493 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi152.hpxjws smithi152 running (85s) 69s ago 85s 14.3M - 17.2.7-268-g6441305d c3a232f785d8 98e18f4a7c6f 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi079.miqczk smithi079 *:9283 running (8m) 63s ago 8m 479M - 17.2.7-268-g6441305d c3a232f785d8 d038b98fa75a 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi152.flyowb smithi152 *:8443,9283 running (5m) 69s ago 5m 433M - 17.2.7-268-g6441305d c3a232f785d8 92e644b493a7 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi079 smithi079 running (8m) 63s ago 8m 48.2M 2048M 17.2.7-268-g6441305d c3a232f785d8 0aba73c41ca9 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi152 smithi152 running (5m) 69s ago 5m 38.6M 2048M 17.2.7-268-g6441305d c3a232f785d8 33474ce60d80 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi079 smithi079 *:9100 running (7m) 63s ago 7m 15.9M - 1.5.0 0da6a335fe13 13236a399b50 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi152 smithi152 *:9100 running (5m) 69s ago 5m 15.4M - 1.5.0 0da6a335fe13 bc329f5a11bd 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (5m) 63s ago 5m 45.6M 4288M 17.2.7-268-g6441305d c3a232f785d8 07dc267fdcd6 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (4m) 63s ago 4m 45.7M 4288M 17.2.7-268-g6441305d c3a232f785d8 76e91cff255f 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (4m) 63s ago 4m 45.9M 4288M 17.2.7-268-g6441305d c3a232f785d8 1fd04b4d8cd2 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi152 running (3m) 69s ago 3m 43.7M 5312M 17.2.7-268-g6441305d c3a232f785d8 51d81c3e87b4 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi152 running (3m) 69s ago 3m 46.3M 5312M 17.2.7-268-g6441305d c3a232f785d8 df8e00ca007c 2024-01-19T01:22:33.494 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi152 running (2m) 69s ago 2m 41.5M 5312M 17.2.7-268-g6441305d c3a232f785d8 81f6d0444b64 2024-01-19T01:22:33.495 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.smithi079 smithi079 *:9095 running (5m) 63s ago 6m 59.0M - 2.43.0 a07b618ecd1d bbe57c5d745e 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb858069030 msgr2=0x7fb85806b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb858069030 0x7fb85806b4e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb864004600 tx=0x7fb86400b040 comp rx=0 tx=0).stop 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fbaf0 msgr2=0x7fb87010c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fbaf0 0x7fb87010c3f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb860004890 tx=0x7fb860009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 shutdown_connections 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb858069030 0x7fb85806b4e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8700fbaf0 0x7fb87010c3f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.496 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 --2- 172.21.15.79:0/455145405 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb8700fcd00 0x7fb87010c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 >> 172.21.15.79:0/455145405 conn(0x7fb8700f7190 msgr2=0x7fb8700fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:33.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 shutdown_connections 2024-01-19T01:22:33.497 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.495+0000 7fb876353700 1 -- 172.21.15.79:0/455145405 wait complete. 2024-01-19T01:22:33.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 -- 172.21.15.79:0/3669832431 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 msgr2=0x7f31ac0ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3669832431 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 0x7f31ac0ff7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f31a8004940 tx=0x7f31a8009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:33.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 -- 172.21.15.79:0/3669832431 shutdown_connections 2024-01-19T01:22:33.606 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3669832431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 0x7f31ac1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3669832431 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 0x7f31ac0ff7a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 -- 172.21.15.79:0/3669832431 >> 172.21.15.79:0/3669832431 conn(0x7f31ac0f71b0 msgr2=0x7f31ac0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:33.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 -- 172.21.15.79:0/3669832431 shutdown_connections 2024-01-19T01:22:33.607 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.605+0000 7f31b41ca700 1 -- 172.21.15.79:0/3669832431 wait complete. 2024-01-19T01:22:33.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b41ca700 1 Processor -- start 2024-01-19T01:22:33.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b41ca700 1 -- start start 2024-01-19T01:22:33.608 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b41ca700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 0x7f31ac0fd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b41ca700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 0x7f31ac0fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b41ca700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac111210 con 0x7f31ac066920 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b41ca700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac111380 con 0x7f31ac0ffd70 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b1f66700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 0x7f31ac0fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b1765700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 0x7f31ac0fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b1765700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 0x7f31ac0fb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.79:42068/0 (socket says 172.21.15.79:42068) 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.606+0000 7f31b1765700 1 -- 172.21.15.79:0/3534462190 learned_addr learned my addr 172.21.15.79:0/3534462190 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31b1765700 1 -- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 msgr2=0x7f31ac0fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31b1765700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 0x7f31ac0fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.609 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31b1765700 1 -- 172.21.15.79:0/3534462190 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8005040 con 0x7f31ac0ffd70 2024-01-19T01:22:33.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31b1765700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 0x7f31ac0fb6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f319c00eb40 tx=0x7f319c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:33.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31a27fc700 1 -- 172.21.15.79:0/3534462190 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f319c007cc0 con 0x7f31ac0ffd70 2024-01-19T01:22:33.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac0fbca0 con 0x7f31ac0ffd70 2024-01-19T01:22:33.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31a27fc700 1 -- 172.21.15.79:0/3534462190 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f319c00f040 con 0x7f31ac0ffd70 2024-01-19T01:22:33.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.607+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac0fc1e0 con 0x7f31ac0ffd70 2024-01-19T01:22:33.610 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.608+0000 7f31a27fc700 1 -- 172.21.15.79:0/3534462190 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f319c00ca30 con 0x7f31ac0ffd70 2024-01-19T01:22:33.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.608+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31900052e0 con 0x7f31ac0ffd70 2024-01-19T01:22:33.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.609+0000 7f31a27fc700 1 -- 172.21.15.79:0/3534462190 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f319c00cb90 con 0x7f31ac0ffd70 2024-01-19T01:22:33.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.610+0000 7f31a27fc700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3198068f60 0x7f319806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:33.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.610+0000 7f31a27fc700 1 -- 172.21.15.79:0/3534462190 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f319c0869a0 con 0x7f31ac0ffd70 2024-01-19T01:22:33.611 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.610+0000 7f31b1f66700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3198068f60 0x7f319806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:33.612 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.611+0000 7f31b1f66700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3198068f60 0x7f319806b410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f31a800b0d0 tx=0x7f31a8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:33.616 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.615+0000 7f31a27fc700 1 -- 172.21.15.79:0/3534462190 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f319c056790 con 0x7f31ac0ffd70 2024-01-19T01:22:33.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.904+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3190005da0 con 0x7f31ac0ffd70 2024-01-19T01:22:33.905 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.905+0000 7f31a27fc700 1 -- 172.21.15.79:0/3534462190 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f319c056320 con 0x7f31ac0ffd70 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 6 2024-01-19T01:22:33.906 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:33.907 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-01-19T01:22:33.907 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 4 2024-01-19T01:22:33.907 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:22:33.907 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-01-19T01:22:33.907 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 14 2024-01-19T01:22:33.907 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-01-19T01:22:33.907 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-01-19T01:22:33.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3198068f60 msgr2=0x7f319806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3198068f60 0x7f319806b410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f31a800b0d0 tx=0x7f31a8009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:33.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 msgr2=0x7f31ac0fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:33.908 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 0x7f31ac0fb6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f319c00eb40 tx=0x7f319c00b040 comp rx=0 tx=0).stop 2024-01-19T01:22:33.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 shutdown_connections 2024-01-19T01:22:33.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f3198068f60 0x7f319806b410 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f31ac066920 0x7f31ac0fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 --2- 172.21.15.79:0/3534462190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31ac0ffd70 0x7f31ac0fb6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:33.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 >> 172.21.15.79:0/3534462190 conn(0x7f31ac0f71b0 msgr2=0x7f31ac0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:33.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 shutdown_connections 2024-01-19T01:22:33.909 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:33.907+0000 7f31b41ca700 1 -- 172.21.15.79:0/3534462190 wait complete. 2024-01-19T01:22:34.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.012+0000 7f6512628700 1 -- 172.21.15.79:0/61932295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 msgr2=0x7f650c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.012+0000 7f6512628700 1 --2- 172.21.15.79:0/61932295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 0x7f650c0ffb20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f64fc004940 tx=0x7f64fc009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:34.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.012+0000 7f6512628700 1 -- 172.21.15.79:0/61932295 shutdown_connections 2024-01-19T01:22:34.013 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.012+0000 7f6512628700 1 --2- 172.21.15.79:0/61932295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 0x7f650c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.014 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:33 smithi079 ceph-mon[28374]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.012+0000 7f6512628700 1 --2- 172.21.15.79:0/61932295 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 0x7f650c0ffb20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.012+0000 7f6512628700 1 -- 172.21.15.79:0/61932295 >> 172.21.15.79:0/61932295 conn(0x7f650c0f71c0 msgr2=0x7f650c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.013+0000 7f6512628700 1 -- 172.21.15.79:0/61932295 shutdown_connections 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.013+0000 7f6512628700 1 -- 172.21.15.79:0/61932295 wait complete. 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.013+0000 7f6512628700 1 Processor -- start 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.013+0000 7f6512628700 1 -- start start 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f6512628700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 0x7f650c0fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.015 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f6512628700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 0x7f650c0fb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f6512628700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c0fbc50 con 0x7f650c0fd740 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f6512628700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c0fbdc0 con 0x7f650c100060 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650b7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 0x7f650c0fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650b7fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 0x7f650c0fb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.79:42084/0 (socket says 172.21.15.79:42084) 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650b7fe700 1 -- 172.21.15.79:0/395396662 learned_addr learned my addr 172.21.15.79:0/395396662 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650bfff700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 0x7f650c0fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650b7fe700 1 -- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 msgr2=0x7f650c0fd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650b7fe700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 0x7f650c0fd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650b7fe700 1 -- 172.21.15.79:0/395396662 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc005040 con 0x7f650c100060 2024-01-19T01:22:34.016 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.014+0000 7f650b7fe700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 0x7f650c0fb650 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f650000eb70 tx=0x7f650000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:34.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.015+0000 7f6508ff9700 1 -- 172.21.15.79:0/395396662 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6500007cc0 con 0x7f650c100060 2024-01-19T01:22:34.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.015+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c0fc040 con 0x7f650c100060 2024-01-19T01:22:34.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.015+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c115500 con 0x7f650c100060 2024-01-19T01:22:34.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.015+0000 7f6508ff9700 1 -- 172.21.15.79:0/395396662 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f650000f040 con 0x7f650c100060 2024-01-19T01:22:34.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.015+0000 7f6508ff9700 1 -- 172.21.15.79:0/395396662 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f650000c9f0 con 0x7f650c100060 2024-01-19T01:22:34.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.016+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c04bb80 con 0x7f650c100060 2024-01-19T01:22:34.017 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.016+0000 7f6508ff9700 1 -- 172.21.15.79:0/395396662 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f65000090d0 con 0x7f650c100060 2024-01-19T01:22:34.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.017+0000 7f6508ff9700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f64f4068f60 0x7f64f406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.017+0000 7f6508ff9700 1 -- 172.21.15.79:0/395396662 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f65000868c0 con 0x7f650c100060 2024-01-19T01:22:34.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.017+0000 7f650bfff700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f64f4068f60 0x7f64f406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.018 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.018+0000 7f650bfff700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f64f4068f60 0x7f64f406b410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f64fc013680 tx=0x7f64fc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:34.025 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.025+0000 7f6508ff9700 1 -- 172.21.15.79:0/395396662 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6500056770 con 0x7f650c100060 2024-01-19T01:22:34.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:33 smithi152 ceph-mon[35694]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.291+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f650c1055b0 con 0x7f650c100060 2024-01-19T01:22:34.292 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.291+0000 7f6508ff9700 1 -- 172.21.15.79:0/395396662 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1809 (secure 0 0 0) 0x7f6500019730 con 0x7f650c100060 2024-01-19T01:22:34.292 INFO:teuthology.orchestra.run.smithi079.stdout:e15 2024-01-19T01:22:34.292 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 15 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-01-19T01:21:01.585027+0000 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-01-19T01:21:18.569403+0000 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 60 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-01-19T01:22:34.293 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 46 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 1 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:in 0 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:up {0=14472} 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data enabled 2024-01-19T01:22:34.294 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 1 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.qxcrcz{0:14472} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout:Standby daemons: 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.cwngbt{-1:14484} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.79:6826/2748689961,v1:172.21.15.79:6827/2748689961] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.hpxjws{-1:24255} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:34.295 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.btzecy{-1:24265} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f64f4068f60 msgr2=0x7f64f406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f64f4068f60 0x7f64f406b410 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f64fc013680 tx=0x7f64fc009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 msgr2=0x7f650c0fb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 0x7f650c0fb650 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f650000eb70 tx=0x7f650000b040 comp rx=0 tx=0).stop 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 shutdown_connections 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f64f4068f60 0x7f64f406b410 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f650c0fd740 0x7f650c0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 --2- 172.21.15.79:0/395396662 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f650c100060 0x7f650c0fb650 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 >> 172.21.15.79:0/395396662 conn(0x7f650c0f71c0 msgr2=0x7f650c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:34.296 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 shutdown_connections 2024-01-19T01:22:34.297 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.294+0000 7f6512628700 1 -- 172.21.15.79:0/395396662 wait complete. 2024-01-19T01:22:34.297 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 15 2024-01-19T01:22:34.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 -- 172.21.15.79:0/1167000975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964066900 msgr2=0x7fb964101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 --2- 172.21.15.79:0/1167000975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964066900 0x7fb964101bc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb954004940 tx=0x7fb954009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:34.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 -- 172.21.15.79:0/1167000975 shutdown_connections 2024-01-19T01:22:34.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 --2- 172.21.15.79:0/1167000975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb964102100 0x7fb9641044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 --2- 172.21.15.79:0/1167000975 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964066900 0x7fb964101bc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 -- 172.21.15.79:0/1167000975 >> 172.21.15.79:0/1167000975 conn(0x7fb9640f71d0 msgr2=0x7fb9640f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:34.406 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 -- 172.21.15.79:0/1167000975 shutdown_connections 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.400+0000 7fb96a182700 1 -- 172.21.15.79:0/1167000975 wait complete. 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.401+0000 7fb96a182700 1 Processor -- start 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.401+0000 7fb96a182700 1 -- start start 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.401+0000 7fb96a182700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb964066900 0x7fb96410f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.401+0000 7fb96a182700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964102100 0x7fb96410fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb96a182700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb964110220 con 0x7fb964102100 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb962ffd700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964102100 0x7fb96410fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb9637fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb964066900 0x7fb96410f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb9637fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb964066900 0x7fb96410f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.79:42096/0 (socket says 172.21.15.79:42096) 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb9637fe700 1 -- 172.21.15.79:0/4101170318 learned_addr learned my addr 172.21.15.79:0/4101170318 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb964110ac0 con 0x7fb964066900 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb962ffd700 1 -- 172.21.15.79:0/4101170318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb964066900 msgr2=0x7fb96410f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb962ffd700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb964066900 0x7fb96410f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb962ffd700 1 -- 172.21.15.79:0/4101170318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb954005040 con 0x7fb964102100 2024-01-19T01:22:34.407 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.402+0000 7fb962ffd700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964102100 0x7fb96410fc20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb964066640 tx=0x7fb95800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:34.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.403+0000 7fb94bfff700 1 -- 172.21.15.79:0/4101170318 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb958004e90 con 0x7fb964102100 2024-01-19T01:22:34.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.403+0000 7fb94bfff700 1 -- 172.21.15.79:0/4101170318 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb95800f040 con 0x7fb964102100 2024-01-19T01:22:34.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.403+0000 7fb94bfff700 1 -- 172.21.15.79:0/4101170318 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95800c980 con 0x7fb964102100 2024-01-19T01:22:34.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.403+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb964110d40 con 0x7fb964102100 2024-01-19T01:22:34.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.404+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb964111280 con 0x7fb964102100 2024-01-19T01:22:34.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.404+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96404bb80 con 0x7fb964102100 2024-01-19T01:22:34.408 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.405+0000 7fb94bfff700 1 -- 172.21.15.79:0/4101170318 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb958007cc0 con 0x7fb964102100 2024-01-19T01:22:34.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.406+0000 7fb94bfff700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb94c068f70 0x7fb94c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.406+0000 7fb94bfff700 1 -- 172.21.15.79:0/4101170318 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb958052ec0 con 0x7fb964102100 2024-01-19T01:22:34.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.406+0000 7fb9637fe700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb94c068f70 0x7fb94c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.409 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.406+0000 7fb9637fe700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb94c068f70 0x7fb94c06b420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb95400b0d0 tx=0x7fb954009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:34.411 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.410+0000 7fb94bfff700 1 -- 172.21.15.79:0/4101170318 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb958056580 con 0x7fb964102100 2024-01-19T01:22:34.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.621+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb964102d60 con 0x7fb94c068f70 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.623+0000 7fb94bfff700 1 -- 172.21.15.79:0/4101170318 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fb964102d60 con 0x7fb94c068f70 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7", 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stdout: "in_progress": true, 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stdout: "which": "Upgrading all daemon types on all hosts", 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stdout: "services_complete": [], 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stdout: "progress": "0/21 daemons upgraded", 2024-01-19T01:22:34.624 INFO:teuthology.orchestra.run.smithi079.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7 image on host smithi152", 2024-01-19T01:22:34.625 INFO:teuthology.orchestra.run.smithi079.stdout: "is_paused": false 2024-01-19T01:22:34.625 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-01-19T01:22:34.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.625+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb94c068f70 msgr2=0x7fb94c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.625+0000 7fb96a182700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb94c068f70 0x7fb94c06b420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb95400b0d0 tx=0x7fb954009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.625+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964102100 msgr2=0x7fb96410fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.625+0000 7fb96a182700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964102100 0x7fb96410fc20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb964066640 tx=0x7fb95800b040 comp rx=0 tx=0).stop 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.626+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 shutdown_connections 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.626+0000 7fb96a182700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fb94c068f70 0x7fb94c06b420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.626+0000 7fb96a182700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb964066900 0x7fb96410f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.626+0000 7fb96a182700 1 --2- 172.21.15.79:0/4101170318 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fb964102100 0x7fb96410fc20 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb964066640 tx=0x7fb95800b040 comp rx=0 tx=0).stop 2024-01-19T01:22:34.627 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.626+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 >> 172.21.15.79:0/4101170318 conn(0x7fb9640f71d0 msgr2=0x7fb9640f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:34.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.626+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 shutdown_connections 2024-01-19T01:22:34.628 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.626+0000 7fb96a182700 1 -- 172.21.15.79:0/4101170318 wait complete. 2024-01-19T01:22:34.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 -- 172.21.15.79:0/3940391779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7544100050 msgr2=0x7f7544102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3940391779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7544100050 0x7f7544102430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7534004940 tx=0x7f7534009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:34.731 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 -- 172.21.15.79:0/3940391779 shutdown_connections 2024-01-19T01:22:34.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3940391779 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7544100050 0x7f7544102430 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3940391779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f75440fd730 0x7f75440ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 -- 172.21.15.79:0/3940391779 >> 172.21.15.79:0/3940391779 conn(0x7f75440f71b0 msgr2=0x7f75440f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:34.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 -- 172.21.15.79:0/3940391779 shutdown_connections 2024-01-19T01:22:34.732 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.730+0000 7f754bd0a700 1 -- 172.21.15.79:0/3940391779 wait complete. 2024-01-19T01:22:34.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f754bd0a700 1 Processor -- start 2024-01-19T01:22:34.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f754bd0a700 1 -- start start 2024-01-19T01:22:34.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f754bd0a700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75440fd730 0x7f754410d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.733 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f754bd0a700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7544100050 0x7f754410da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f754bd0a700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754410e070 con 0x7f75440fd730 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f754bd0a700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754410e910 con 0x7f7544100050 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f7549aa6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75440fd730 0x7f754410d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f7549aa6700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75440fd730 0x7f754410d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:55574/0 (socket says 172.21.15.79:55574) 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.731+0000 7f7549aa6700 1 -- 172.21.15.79:0/3540038584 learned_addr learned my addr 172.21.15.79:0/3540038584 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f75492a5700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7544100050 0x7f754410da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f7549aa6700 1 -- 172.21.15.79:0/3540038584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7544100050 msgr2=0x7f754410da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f7549aa6700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7544100050 0x7f754410da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f7549aa6700 1 -- 172.21.15.79:0/3540038584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7534005040 con 0x7f75440fd730 2024-01-19T01:22:34.734 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f7549aa6700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75440fd730 0x7f754410d530 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7540002a50 tx=0x7f75400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:34.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f753a7fc700 1 -- 172.21.15.79:0/3540038584 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7540007cc0 con 0x7f75440fd730 2024-01-19T01:22:34.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f753a7fc700 1 -- 172.21.15.79:0/3540038584 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f754000f040 con 0x7f75440fd730 2024-01-19T01:22:34.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f753a7fc700 1 -- 172.21.15.79:0/3540038584 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f754000d9c0 con 0x7f75440fd730 2024-01-19T01:22:34.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754410ebf0 con 0x7f75440fd730 2024-01-19T01:22:34.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.732+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754410f130 con 0x7f75440fd730 2024-01-19T01:22:34.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.734+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75441075c0 con 0x7f75440fd730 2024-01-19T01:22:34.735 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.734+0000 7f753a7fc700 1 -- 172.21.15.79:0/3540038584 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f754000db40 con 0x7f75440fd730 2024-01-19T01:22:34.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.735+0000 7f753a7fc700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7530068f60 0x7f753006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:22:34.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.735+0000 7f753a7fc700 1 -- 172.21.15.79:0/3540038584 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7540086a10 con 0x7f75440fd730 2024-01-19T01:22:34.736 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.735+0000 7f75492a5700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7530068f60 0x7f753006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:22:34.737 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.736+0000 7f75492a5700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7530068f60 0x7f753006b410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f753400b3b0 tx=0x7f7534009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:22:34.740 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:34.740+0000 7f753a7fc700 1 -- 172.21.15.79:0/3540038584 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f75400569d0 con 0x7f75440fd730 2024-01-19T01:22:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:34 smithi079 ceph-mon[28374]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:34 smithi079 ceph-mon[28374]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:22:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:34 smithi079 ceph-mon[28374]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:34 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3534462190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T01:22:34.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:34 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:34.999 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:34 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/395396662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:22:35.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.122+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f754404bb80 con 0x7f75440fd730 2024-01-19T01:22:35.123 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.122+0000 7f753a7fc700 1 -- 172.21.15.79:0/3540038584 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f754000ddf0 con 0x7f75440fd730 2024-01-19T01:22:35.123 INFO:teuthology.orchestra.run.smithi079.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-01-19T01:22:35.123 INFO:teuthology.orchestra.run.smithi079.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-01-19T01:22:35.124 INFO:teuthology.orchestra.run.smithi079.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-01-19T01:22:35.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7530068f60 msgr2=0x7f753006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:35.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7530068f60 0x7f753006b410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f753400b3b0 tx=0x7f7534009e30 comp rx=0 tx=0).stop 2024-01-19T01:22:35.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75440fd730 msgr2=0x7f754410d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:22:35.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75440fd730 0x7f754410d530 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7540002a50 tx=0x7f75400046b0 comp rx=0 tx=0).stop 2024-01-19T01:22:35.126 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 shutdown_connections 2024-01-19T01:22:35.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7530068f60 0x7f753006b410 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:35.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f75440fd730 0x7f754410d530 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:35.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 --2- 172.21.15.79:0/3540038584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7544100050 0x7f754410da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:22:35.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 >> 172.21.15.79:0/3540038584 conn(0x7f75440f71b0 msgr2=0x7f75440f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:22:35.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 shutdown_connections 2024-01-19T01:22:35.127 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:22:35.125+0000 7f754bd0a700 1 -- 172.21.15.79:0/3540038584 wait complete. 2024-01-19T01:22:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:34 smithi152 ceph-mon[35694]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:34 smithi152 ceph-mon[35694]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-01-19T01:22:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:34 smithi152 ceph-mon[35694]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:34 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3534462190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T01:22:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:34 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:35.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:34 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/395396662' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:22:35.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:35 smithi152 ceph-mon[35694]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:35.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:35 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3540038584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-01-19T01:22:36.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:35 smithi079 ceph-mon[28374]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:22:36.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:35 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3540038584' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-01-19T01:22:37.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:36 smithi152 ceph-mon[35694]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:37.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:36 smithi079 ceph-mon[28374]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:38.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:38 smithi079 ceph-mon[28374]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:39.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:38 smithi152 ceph-mon[35694]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:41.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:40 smithi152 ceph-mon[35694]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:41.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:40 smithi079 ceph-mon[28374]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:43.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:42 smithi152 ceph-mon[35694]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:43.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:42 smithi079 ceph-mon[28374]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:43.803 INFO:tasks.workunit.client.1.smithi152.stderr:Updating files: 93% (11289/12104) Updating files: 94% (11378/12104) Updating files: 95% (11499/12104) Updating files: 96% (11620/12104) Updating files: 97% (11741/12104) Updating files: 98% (11862/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-01-19T01:22:44.659 INFO:tasks.workunit.client.1.smithi152.stderr:Note: switching to '0edf41a622739843d6b978b179ff3227b476dd9d'. 2024-01-19T01:22:44.659 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-01-19T01:22:44.659 INFO:tasks.workunit.client.1.smithi152.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-19T01:22:44.659 INFO:tasks.workunit.client.1.smithi152.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-19T01:22:44.659 INFO:tasks.workunit.client.1.smithi152.stderr:state without impacting any branches by switching back to a branch. 2024-01-19T01:22:44.659 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-01-19T01:22:44.659 INFO:tasks.workunit.client.1.smithi152.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-19T01:22:44.660 INFO:tasks.workunit.client.1.smithi152.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-19T01:22:44.660 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-01-19T01:22:44.660 INFO:tasks.workunit.client.1.smithi152.stderr: git switch -c 2024-01-19T01:22:44.660 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-01-19T01:22:44.660 INFO:tasks.workunit.client.1.smithi152.stderr:Or undo this operation with: 2024-01-19T01:22:44.660 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-01-19T01:22:44.660 INFO:tasks.workunit.client.1.smithi152.stderr: git switch - 2024-01-19T01:22:44.661 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-01-19T01:22:44.661 INFO:tasks.workunit.client.1.smithi152.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-19T01:22:44.661 INFO:tasks.workunit.client.1.smithi152.stderr: 2024-01-19T01:22:44.661 INFO:tasks.workunit.client.1.smithi152.stderr:HEAD is now at 0edf41a6227 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2024-01-18-0823 2024-01-19T01:22:44.663 DEBUG:teuthology.orchestra.run.smithi152:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-01-19T01:22:44.729 INFO:tasks.workunit.client.1.smithi152.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-19T01:22:44.732 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-01-19T01:22:44.732 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-19T01:22:44.795 INFO:tasks.workunit.client.0.smithi079.stderr:Note: switching to '0edf41a622739843d6b978b179ff3227b476dd9d'. 2024-01-19T01:22:44.795 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-01-19T01:22:44.795 INFO:tasks.workunit.client.0.smithi079.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr:state without impacting any branches by switching back to a branch. 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr: git switch -c 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr:Or undo this operation with: 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr: git switch - 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-01-19T01:22:44.796 INFO:tasks.workunit.client.0.smithi079.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-19T01:22:44.797 INFO:tasks.workunit.client.0.smithi079.stderr: 2024-01-19T01:22:44.797 INFO:tasks.workunit.client.0.smithi079.stderr:HEAD is now at 0edf41a6227 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2024-01-18-0823 2024-01-19T01:22:44.799 DEBUG:teuthology.orchestra.run.smithi079:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-19T01:22:44.829 INFO:tasks.workunit.client.0.smithi079.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-19T01:22:44.832 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-19T01:22:44.832 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-19T01:22:45.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:44 smithi152 ceph-mon[35694]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:45.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:44 smithi079 ceph-mon[28374]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:45.512 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-19T01:22:45.580 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-19T01:22:45.592 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-19T01:22:45.624 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-01-19T01:22:45.625 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-01-19T01:22:45.625 INFO:tasks.workunit.client.1.smithi152.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-19T01:22:45.663 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-19T01:22:45.673 INFO:tasks.workunit.client.1.smithi152.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-01-19T01:22:45.676 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-01-19T01:22:45.676 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-01-19T01:22:45.705 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-19T01:22:45.706 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-19T01:22:45.706 INFO:tasks.workunit.client.0.smithi079.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-19T01:22:45.732 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-01-19T01:22:45.735 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-01-19T01:22:45.735 DEBUG:teuthology.orchestra.run.smithi152:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0edf41a622739843d6b978b179ff3227b476dd9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-01-19T01:22:45.745 INFO:tasks.workunit.client.0.smithi079.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-19T01:22:45.748 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-01-19T01:22:45.749 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-19T01:22:45.793 INFO:tasks.workunit.client.1.smithi152.stderr:+ mkdir -p fsstress 2024-01-19T01:22:45.794 INFO:tasks.workunit.client.1.smithi152.stderr:+ pushd fsstress 2024-01-19T01:22:45.794 INFO:tasks.workunit.client.1.smithi152.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-01-19T01:22:45.795 INFO:tasks.workunit.client.1.smithi152.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-01-19T01:22:45.804 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-01-19T01:22:45.805 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-01-19T01:22:45.805 DEBUG:teuthology.orchestra.run.smithi079:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0edf41a622739843d6b978b179ff3227b476dd9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-01-19T01:22:45.863 INFO:tasks.workunit.client.0.smithi079.stderr:+ mkdir -p fsstress 2024-01-19T01:22:45.864 INFO:tasks.workunit.client.0.smithi079.stderr:+ pushd fsstress 2024-01-19T01:22:45.865 INFO:tasks.workunit.client.0.smithi079.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-01-19T01:22:45.865 INFO:tasks.workunit.client.0.smithi079.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-01-19T01:22:46.730 INFO:tasks.workunit.client.1.smithi152.stderr:+ tar xzf ltp-full.tgz 2024-01-19T01:22:46.826 INFO:tasks.workunit.client.0.smithi079.stderr:+ tar xzf ltp-full.tgz 2024-01-19T01:22:47.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:46 smithi152 ceph-mon[35694]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:47.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:46 smithi079 ceph-mon[28374]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:48.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:48 smithi079 ceph-mon[28374]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:49.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:48 smithi152 ceph-mon[35694]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-01-19T01:22:50.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:49 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:50.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:49 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:22:51.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:50 smithi152 ceph-mon[35694]: pgmap v229: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-01-19T01:22:51.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:50 smithi079 ceph-mon[28374]: pgmap v229: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-01-19T01:22:53.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:52 smithi152 ceph-mon[35694]: pgmap v230: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-01-19T01:22:53.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:52 smithi079 ceph-mon[28374]: pgmap v230: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s wr, 1 op/s 2024-01-19T01:22:55.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:54 smithi152 ceph-mon[35694]: pgmap v231: 65 pgs: 65 active+clean; 18 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s wr, 77 op/s 2024-01-19T01:22:55.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:54 smithi079 ceph-mon[28374]: pgmap v231: 65 pgs: 65 active+clean; 18 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s wr, 77 op/s 2024-01-19T01:22:56.173 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:55 smithi152 ceph-mon[35694]: pgmap v232: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.4 MiB/s wr, 470 op/s 2024-01-19T01:22:56.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:55 smithi079 ceph-mon[28374]: pgmap v232: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.4 MiB/s wr, 470 op/s 2024-01-19T01:22:58.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:58 smithi152 ceph-mon[35694]: pgmap v233: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.4 MiB/s wr, 470 op/s 2024-01-19T01:22:58.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:58 smithi079 ceph-mon[28374]: pgmap v233: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 5.4 MiB/s wr, 470 op/s 2024-01-19T01:22:59.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:59 smithi079 ceph-mon[28374]: Upgrade: Updating mgr.smithi152.flyowb 2024-01-19T01:22:59.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:59 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:22:59.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:59 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:22:59.748 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:59 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:22:59.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:59 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:22:59.749 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:22:59 smithi079 ceph-mon[28374]: Deploying daemon mgr.smithi152.flyowb on smithi152 2024-01-19T01:22:59.779 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:59 smithi152 ceph-mon[35694]: Upgrade: Updating mgr.smithi152.flyowb 2024-01-19T01:22:59.779 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:59 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:22:59.779 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:59 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi152.flyowb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-19T01:22:59.779 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:59 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-19T01:22:59.779 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:59 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-19T01:22:59.779 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:22:59 smithi152 ceph-mon[35694]: Deploying daemon mgr.smithi152.flyowb on smithi152 2024-01-19T01:23:00.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:00 smithi152 ceph-mon[35694]: pgmap v234: 65 pgs: 65 active+clean; 74 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 487 op/s 2024-01-19T01:23:00.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:00 smithi079 ceph-mon[28374]: pgmap v234: 65 pgs: 65 active+clean; 74 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 487 op/s 2024-01-19T01:23:02.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:02 smithi152 ceph-mon[35694]: pgmap v235: 65 pgs: 65 active+clean; 74 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.1 MiB/s wr, 486 op/s 2024-01-19T01:23:02.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:02 smithi079 ceph-mon[28374]: pgmap v235: 65 pgs: 65 active+clean; 74 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.1 MiB/s wr, 486 op/s 2024-01-19T01:23:04.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:04 smithi152 ceph-mon[35694]: pgmap v236: 65 pgs: 65 active+clean; 77 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 530 op/s 2024-01-19T01:23:04.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:04 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:23:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:04 smithi079 ceph-mon[28374]: pgmap v236: 65 pgs: 65 active+clean; 77 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s wr, 530 op/s 2024-01-19T01:23:04.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:04 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-01-19T01:23:05.261 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.258+0000 7fef38698700 1 -- 172.21.15.79:0/2590449802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fbaf0 msgr2=0x7fef300fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.258+0000 7fef38698700 1 --2- 172.21.15.79:0/2590449802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fbaf0 0x7fef300fbf00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fef2c004940 tx=0x7fef2c009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 -- 172.21.15.79:0/2590449802 shutdown_connections 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 --2- 172.21.15.79:0/2590449802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef300fcd00 0x7fef300fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 --2- 172.21.15.79:0/2590449802 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fbaf0 0x7fef300fbf00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 -- 172.21.15.79:0/2590449802 >> 172.21.15.79:0/2590449802 conn(0x7fef300f71b0 msgr2=0x7fef300f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 -- 172.21.15.79:0/2590449802 shutdown_connections 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 -- 172.21.15.79:0/2590449802 wait complete. 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 Processor -- start 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.259+0000 7fef38698700 1 -- start start 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef38698700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef300fbaf0 0x7fef3010c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef38698700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fcd00 0x7fef3010c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef38698700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3010cfd0 con 0x7fef300fcd00 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef38698700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3010d870 con 0x7fef300fbaf0 2024-01-19T01:23:05.262 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef35c33700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fcd00 0x7fef3010c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef35c33700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fcd00 0x7fef3010c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56270/0 (socket says 172.21.15.79:56270) 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef35c33700 1 -- 172.21.15.79:0/2734248983 learned_addr learned my addr 172.21.15.79:0/2734248983 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef36434700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef300fbaf0 0x7fef3010c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef35c33700 1 -- 172.21.15.79:0/2734248983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef300fbaf0 msgr2=0x7fef3010c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef35c33700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef300fbaf0 0x7fef3010c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef35c33700 1 -- 172.21.15.79:0/2734248983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c005040 con 0x7fef300fcd00 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.260+0000 7fef35c33700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fcd00 0x7fef3010c9d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef2000d990 tx=0x7fef2000a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.261+0000 7fef26ffd700 1 -- 172.21.15.79:0/2734248983 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2000c9b0 con 0x7fef300fcd00 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.261+0000 7fef26ffd700 1 -- 172.21.15.79:0/2734248983 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef20010460 con 0x7fef300fcd00 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.261+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3010daf0 con 0x7fef300fcd00 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.261+0000 7fef26ffd700 1 -- 172.21.15.79:0/2734248983 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2000f5e0 con 0x7fef300fcd00 2024-01-19T01:23:05.263 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.261+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3010e030 con 0x7fef300fcd00 2024-01-19T01:23:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.262+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3004bb80 con 0x7fef300fcd00 2024-01-19T01:23:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.262+0000 7fef26ffd700 1 -- 172.21.15.79:0/2734248983 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fef200105d0 con 0x7fef300fcd00 2024-01-19T01:23:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.263+0000 7fef26ffd700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fef1c068f70 0x7fef1c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.263+0000 7fef26ffd700 1 -- 172.21.15.79:0/2734248983 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fef20087030 con 0x7fef300fcd00 2024-01-19T01:23:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.263+0000 7fef36434700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fef1c068f70 0x7fef1c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:05.265 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.263+0000 7fef36434700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fef1c068f70 0x7fef1c06b420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00b3b0 tx=0x7fef2c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:05.267 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.267+0000 7fef26ffd700 1 -- 172.21.15.79:0/2734248983 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fef20056ec0 con 0x7fef300fcd00 2024-01-19T01:23:05.489 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.488+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef3010e590 con 0x7fef1c068f70 2024-01-19T01:23:05.490 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.489+0000 7fef26ffd700 1 -- 172.21.15.79:0/2734248983 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7fef3010e590 con 0x7fef1c068f70 2024-01-19T01:23:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fef1c068f70 msgr2=0x7fef1c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fef1c068f70 0x7fef1c06b420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00b3b0 tx=0x7fef2c009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fcd00 msgr2=0x7fef3010c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.492 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fcd00 0x7fef3010c9d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef2000d990 tx=0x7fef2000a0f0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 shutdown_connections 2024-01-19T01:23:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7fef1c068f70 0x7fef1c06b420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef300fbaf0 0x7fef3010c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 --2- 172.21.15.79:0/2734248983 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7fef300fcd00 0x7fef3010c9d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 >> 172.21.15.79:0/2734248983 conn(0x7fef300f71b0 msgr2=0x7fef300f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 shutdown_connections 2024-01-19T01:23:05.493 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.491+0000 7fef38698700 1 -- 172.21.15.79:0/2734248983 wait complete. 2024-01-19T01:23:05.505 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-01-19T01:23:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.616+0000 7f40a6f17700 1 -- 172.21.15.79:0/107346792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 msgr2=0x7f40a01021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.616+0000 7f40a6f17700 1 --2- 172.21.15.79:0/107346792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a01021d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f408c004940 tx=0x7f408c009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:05.617 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.616+0000 7f40a6f17700 1 -- 172.21.15.79:0/107346792 shutdown_connections 2024-01-19T01:23:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.616+0000 7f40a6f17700 1 --2- 172.21.15.79:0/107346792 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a01021d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.616+0000 7f40a6f17700 1 --2- 172.21.15.79:0/107346792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40a0066090 0x7f40a00ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.616+0000 7f40a6f17700 1 -- 172.21.15.79:0/107346792 >> 172.21.15.79:0/107346792 conn(0x7f40a00f7190 msgr2=0x7f40a00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.616+0000 7f40a6f17700 1 -- 172.21.15.79:0/107346792 shutdown_connections 2024-01-19T01:23:05.618 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.617+0000 7f40a6f17700 1 -- 172.21.15.79:0/107346792 wait complete. 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.617+0000 7f40a6f17700 1 Processor -- start 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.617+0000 7f40a6f17700 1 -- start start 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f40a6f17700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40a0066090 0x7f40a00fd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f40a6f17700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a00fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f40a6f17700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00fbca0 con 0x7f40a00ffd70 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f40a6f17700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a00fbe10 con 0x7f40a0066090 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a00fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f40a4cb3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40a0066090 0x7f40a00fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a00fb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56286/0 (socket says 172.21.15.79:56286) 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409ffff700 1 -- 172.21.15.79:0/1813662845 learned_addr learned my addr 172.21.15.79:0/1813662845 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:23:05.619 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409ffff700 1 -- 172.21.15.79:0/1813662845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40a0066090 msgr2=0x7f40a00fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409ffff700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40a0066090 0x7f40a00fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409ffff700 1 -- 172.21.15.79:0/1813662845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c005040 con 0x7f40a00ffd70 2024-01-19T01:23:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409ffff700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a00fb6a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f408c004f90 tx=0x7f408c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:05.620 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.618+0000 7f409d7fa700 1 -- 172.21.15.79:0/1813662845 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f408c00e040 con 0x7f40a00ffd70 2024-01-19T01:23:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.619+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a00fc090 con 0x7f40a00ffd70 2024-01-19T01:23:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.619+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a0115540 con 0x7f40a00ffd70 2024-01-19T01:23:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.619+0000 7f409d7fa700 1 -- 172.21.15.79:0/1813662845 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f408c00bc50 con 0x7f40a00ffd70 2024-01-19T01:23:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.619+0000 7f409d7fa700 1 -- 172.21.15.79:0/1813662845 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f408c01c630 con 0x7f40a00ffd70 2024-01-19T01:23:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.620+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40840052e0 con 0x7f40a00ffd70 2024-01-19T01:23:05.621 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.620+0000 7f409d7fa700 1 -- 172.21.15.79:0/1813662845 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f408c01c790 con 0x7f40a00ffd70 2024-01-19T01:23:05.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.620+0000 7f409d7fa700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4090068f60 0x7f409006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:05.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.621+0000 7f409d7fa700 1 -- 172.21.15.79:0/1813662845 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f408c088020 con 0x7f40a00ffd70 2024-01-19T01:23:05.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.621+0000 7f40a4cb3700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4090068f60 0x7f409006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:05.622 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.621+0000 7f40a4cb3700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4090068f60 0x7f409006b410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40a00fcfb0 tx=0x7f409400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:05.626 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.625+0000 7f409d7fa700 1 -- 172.21.15.79:0/1813662845 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f408c057fd0 con 0x7f40a00ffd70 2024-01-19T01:23:05.938 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.937+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4084000bf0 con 0x7f4090068f60 2024-01-19T01:23:05.939 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.938+0000 7f409d7fa700 1 -- 172.21.15.79:0/1813662845 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f4084000bf0 con 0x7f4090068f60 2024-01-19T01:23:05.941 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.940+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4090068f60 msgr2=0x7f409006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.940+0000 7f40a6f17700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4090068f60 0x7f409006b410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40a00fcfb0 tx=0x7f409400b040 comp rx=0 tx=0).stop 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.940+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 msgr2=0x7f40a00fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.940+0000 7f40a6f17700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a00fb6a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f408c004f90 tx=0x7f408c009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.940+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 shutdown_connections 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.941+0000 7f40a6f17700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f4090068f60 0x7f409006b410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.941+0000 7f40a6f17700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40a0066090 0x7f40a00fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.941+0000 7f40a6f17700 1 --2- 172.21.15.79:0/1813662845 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f40a00ffd70 0x7f40a00fb6a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.941+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 >> 172.21.15.79:0/1813662845 conn(0x7f40a00f7190 msgr2=0x7f40a00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.941+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 shutdown_connections 2024-01-19T01:23:05.942 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:05.941+0000 7f40a6f17700 1 -- 172.21.15.79:0/1813662845 wait complete. 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 -- 172.21.15.79:0/2020662787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200faed0 msgr2=0x7f43200fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 --2- 172.21.15.79:0/2020662787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200faed0 0x7f43200fb2e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f431c004940 tx=0x7f431c009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 -- 172.21.15.79:0/2020662787 shutdown_connections 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 --2- 172.21.15.79:0/2020662787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43200fdf60 0x7f43200fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 --2- 172.21.15.79:0/2020662787 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200faed0 0x7f43200fb2e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 -- 172.21.15.79:0/2020662787 >> 172.21.15.79:0/2020662787 conn(0x7f43200f6950 msgr2=0x7f43200f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 -- 172.21.15.79:0/2020662787 shutdown_connections 2024-01-19T01:23:06.068 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 -- 172.21.15.79:0/2020662787 wait complete. 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 Processor -- start 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.067+0000 7f4328795700 1 -- start start 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4328795700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43200faed0 0x7f432010d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4328795700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200fdf60 0x7f432010db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4328795700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432010e1d0 con 0x7f43200fdf60 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4328795700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432010ea10 con 0x7f43200faed0 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4325d30700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200fdf60 0x7f432010db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4325d30700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200fdf60 0x7f432010db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56312/0 (socket says 172.21.15.79:56312) 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4325d30700 1 -- 172.21.15.79:0/2044045451 learned_addr learned my addr 172.21.15.79:0/2044045451 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4326531700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43200faed0 0x7f432010d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4325d30700 1 -- 172.21.15.79:0/2044045451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43200faed0 msgr2=0x7f432010d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4325d30700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43200faed0 0x7f432010d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4325d30700 1 -- 172.21.15.79:0/2044045451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c005040 con 0x7f43200fdf60 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4325d30700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200fdf60 0x7f432010db40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f431000b790 tx=0x7f4310009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:06.070 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.068+0000 7f4316ffd700 1 -- 172.21.15.79:0/2044045451 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f431000f6f0 con 0x7f43200fdf60 2024-01-19T01:23:06.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.069+0000 7f4316ffd700 1 -- 172.21.15.79:0/2044045451 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f431000fd00 con 0x7f43200fdf60 2024-01-19T01:23:06.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.069+0000 7f4316ffd700 1 -- 172.21.15.79:0/2044045451 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43100103a0 con 0x7f43200fdf60 2024-01-19T01:23:06.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.069+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432010ecf0 con 0x7f43200fdf60 2024-01-19T01:23:06.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.069+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432010f230 con 0x7f43200fdf60 2024-01-19T01:23:06.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.069+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4320107670 con 0x7f43200fdf60 2024-01-19T01:23:06.071 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.070+0000 7f4316ffd700 1 -- 172.21.15.79:0/2044045451 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f431000f850 con 0x7f43200fdf60 2024-01-19T01:23:06.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.071+0000 7f4316ffd700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f430c068f60 0x7f430c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.071+0000 7f4316ffd700 1 -- 172.21.15.79:0/2044045451 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4310086bd0 con 0x7f43200fdf60 2024-01-19T01:23:06.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.071+0000 7f4326531700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f430c068f60 0x7f430c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.072 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.071+0000 7f4326531700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f430c068f60 0x7f430c06b410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f431c00b3b0 tx=0x7f431c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:06.078 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.077+0000 7f4316ffd700 1 -- 172.21.15.79:0/2044045451 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4310056ae0 con 0x7f43200fdf60 2024-01-19T01:23:06.331 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.330+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f432010f5b0 con 0x7f430c068f60 2024-01-19T01:23:06.341 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.340+0000 7f4316ffd700 1 -- 172.21.15.79:0/2044045451 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f432010f5b0 con 0x7f430c068f60 2024-01-19T01:23:06.341 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-01-19T01:23:06.341 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.smithi079 smithi079 *:9093,9094 running (6m) 96s ago 7m 25.7M - 0.25.0 c8568f914cd2 e31a5a73bdf6 2024-01-19T01:23:06.341 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi079 smithi079 running (7m) 96s ago 7m 6665k - 17.2.7-268-g6441305d c3a232f785d8 3b02e2e6b9be 2024-01-19T01:23:06.341 INFO:teuthology.orchestra.run.smithi079.stdout:crash.smithi152 smithi152 running (6m) 102s ago 6m 6660k - 17.2.7-268-g6441305d c3a232f785d8 b47c25b14963 2024-01-19T01:23:06.341 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.smithi079 smithi079 *:3000 running (5m) 96s ago 7m 85.2M - 9.4.7 954c08fa6188 551c78ecdb2b 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.cwngbt smithi079 running (2m) 96s ago 2m 10.4M - 17.2.7-268-g6441305d c3a232f785d8 d9755e8a61f6 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi079.qxcrcz smithi079 running (115s) 96s ago 115s 16.5M - 17.2.7-268-g6441305d c3a232f785d8 6bb941a74e26 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi152.btzecy smithi152 running (113s) 102s ago 112s 12.0M - 17.2.7-268-g6441305d c3a232f785d8 8bef9d65c58e 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mds.cephfs.smithi152.hpxjws smithi152 running (118s) 102s ago 118s 14.3M - 17.2.7-268-g6441305d c3a232f785d8 98e18f4a7c6f 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi079.miqczk smithi079 *:9283 running (8m) 96s ago 8m 479M - 17.2.7-268-g6441305d c3a232f785d8 d038b98fa75a 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.smithi152.flyowb smithi152 *:8443,9283 running (6m) 102s ago 6m 433M - 17.2.7-268-g6441305d c3a232f785d8 92e644b493a7 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi079 smithi079 running (8m) 96s ago 8m 48.2M 2048M 17.2.7-268-g6441305d c3a232f785d8 0aba73c41ca9 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:mon.smithi152 smithi152 running (6m) 102s ago 6m 38.6M 2048M 17.2.7-268-g6441305d c3a232f785d8 33474ce60d80 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi079 smithi079 *:9100 running (7m) 96s ago 7m 15.9M - 1.5.0 0da6a335fe13 13236a399b50 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.smithi152 smithi152 *:9100 running (6m) 102s ago 6m 15.4M - 1.5.0 0da6a335fe13 bc329f5a11bd 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (5m) 96s ago 5m 45.6M 4288M 17.2.7-268-g6441305d c3a232f785d8 07dc267fdcd6 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (5m) 96s ago 5m 45.7M 4288M 17.2.7-268-g6441305d c3a232f785d8 76e91cff255f 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (4m) 96s ago 4m 45.9M 4288M 17.2.7-268-g6441305d c3a232f785d8 1fd04b4d8cd2 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi152 running (4m) 102s ago 4m 43.7M 5312M 17.2.7-268-g6441305d c3a232f785d8 51d81c3e87b4 2024-01-19T01:23:06.342 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi152 running (3m) 102s ago 3m 46.3M 5312M 17.2.7-268-g6441305d c3a232f785d8 df8e00ca007c 2024-01-19T01:23:06.343 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi152 running (3m) 102s ago 3m 41.5M 5312M 17.2.7-268-g6441305d c3a232f785d8 81f6d0444b64 2024-01-19T01:23:06.343 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.smithi079 smithi079 *:9095 running (5m) 96s ago 6m 59.0M - 2.43.0 a07b618ecd1d bbe57c5d745e 2024-01-19T01:23:06.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f430c068f60 msgr2=0x7f430c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f430c068f60 0x7f430c06b410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f431c00b3b0 tx=0x7f431c009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:06.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200fdf60 msgr2=0x7f432010db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.343 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200fdf60 0x7f432010db40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f431000b790 tx=0x7f4310009250 comp rx=0 tx=0).stop 2024-01-19T01:23:06.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 shutdown_connections 2024-01-19T01:23:06.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f430c068f60 0x7f430c06b410 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43200faed0 0x7f432010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 --2- 172.21.15.79:0/2044045451 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f43200fdf60 0x7f432010db40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.342+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 >> 172.21.15.79:0/2044045451 conn(0x7f43200f6950 msgr2=0x7f43200f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:06.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.343+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 shutdown_connections 2024-01-19T01:23:06.344 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.343+0000 7f4328795700 1 -- 172.21.15.79:0/2044045451 wait complete. 2024-01-19T01:23:06.457 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 -- 172.21.15.79:0/3321249296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19780fd730 msgr2=0x7f19780ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 --2- 172.21.15.79:0/3321249296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19780fd730 0x7f19780ffb10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1968004940 tx=0x7f1968009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:06.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 -- 172.21.15.79:0/3321249296 shutdown_connections 2024-01-19T01:23:06.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 --2- 172.21.15.79:0/3321249296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1978100050 0x7f1978102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 --2- 172.21.15.79:0/3321249296 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f19780fd730 0x7f19780ffb10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 -- 172.21.15.79:0/3321249296 >> 172.21.15.79:0/3321249296 conn(0x7f19780f7190 msgr2=0x7f19780f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:06.458 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 -- 172.21.15.79:0/3321249296 shutdown_connections 2024-01-19T01:23:06.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.456+0000 7f197d850700 1 -- 172.21.15.79:0/3321249296 wait complete. 2024-01-19T01:23:06.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f197d850700 1 Processor -- start 2024-01-19T01:23:06.459 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f197d850700 1 -- start start 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f197d850700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19780fd730 0x7f197810d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f197d850700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1978100050 0x7f197810da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f197d850700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197810e040 con 0x7f1978100050 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f197d850700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197810e8e0 con 0x7f19780fd730 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f196ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1978100050 0x7f197810da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f196ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1978100050 0x7f197810da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56334/0 (socket says 172.21.15.79:56334) 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f196ffff700 1 -- 172.21.15.79:0/2276391616 learned_addr learned my addr 172.21.15.79:0/2276391616 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:23:06.460 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.457+0000 7f1976ffd700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19780fd730 0x7f197810d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f196ffff700 1 -- 172.21.15.79:0/2276391616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19780fd730 msgr2=0x7f197810d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f196ffff700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19780fd730 0x7f197810d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f196ffff700 1 -- 172.21.15.79:0/2276391616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968005040 con 0x7f1978100050 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f196ffff700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1978100050 0x7f197810da40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1960002a50 tx=0x7f19600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f196f7fe700 1 -- 172.21.15.79:0/2276391616 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1960007cc0 con 0x7f1978100050 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197810ebc0 con 0x7f1978100050 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f196f7fe700 1 -- 172.21.15.79:0/2276391616 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f196000f040 con 0x7f1978100050 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f196f7fe700 1 -- 172.21.15.79:0/2276391616 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f196000d9c0 con 0x7f1978100050 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.458+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197810f100 con 0x7f1978100050 2024-01-19T01:23:06.461 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.459+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197804bb80 con 0x7f1978100050 2024-01-19T01:23:06.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.460+0000 7f196f7fe700 1 -- 172.21.15.79:0/2276391616 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f196000db20 con 0x7f1978100050 2024-01-19T01:23:06.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.460+0000 7f196f7fe700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1964068f60 0x7f196406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.460+0000 7f196f7fe700 1 -- 172.21.15.79:0/2276391616 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1960086a30 con 0x7f1978100050 2024-01-19T01:23:06.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.460+0000 7f1976ffd700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1964068f60 0x7f196406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.462 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.461+0000 7f1976ffd700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1964068f60 0x7f196406b410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f196800b3b0 tx=0x7f1968009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:06.466 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.466+0000 7f196f7fe700 1 -- 172.21.15.79:0/2276391616 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1960056940 con 0x7f1978100050 2024-01-19T01:23:06.798 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.797+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1978105580 con 0x7f1978100050 2024-01-19T01:23:06.799 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.798+0000 7f196f7fe700 1 -- 172.21.15.79:0/2276391616 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f19600564d0 con 0x7f1978100050 2024-01-19T01:23:06.799 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-01-19T01:23:06.799 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-01-19T01:23:06.799 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T01:23:06.799 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:23:06.799 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-01-19T01:23:06.799 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 2 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 6 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: "mds": { 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 4 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 17.2.7-268-g6441305d (6441305d08929b8a72c0322a5c268c64b4a99c65) quincy (stable)": 14 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-01-19T01:23:06.800 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-01-19T01:23:06.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1964068f60 msgr2=0x7f196406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1964068f60 0x7f196406b410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f196800b3b0 tx=0x7f1968009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:06.801 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1978100050 msgr2=0x7f197810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1978100050 0x7f197810da40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1960002a50 tx=0x7f19600046b0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 shutdown_connections 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f1964068f60 0x7f196406b410 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19780fd730 0x7f197810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 --2- 172.21.15.79:0/2276391616 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f1978100050 0x7f197810da40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.800+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 >> 172.21.15.79:0/2276391616 conn(0x7f19780f7190 msgr2=0x7f19780f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.801+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 shutdown_connections 2024-01-19T01:23:06.802 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.801+0000 7f197d850700 1 -- 172.21.15.79:0/2276391616 wait complete. 2024-01-19T01:23:06.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:06 smithi152 ceph-mon[35694]: pgmap v237: 65 pgs: 65 active+clean; 119 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 9.0 MiB/s wr, 974 op/s 2024-01-19T01:23:06.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:06 smithi152 ceph-mon[35694]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:06.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.931+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3776727441 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 msgr2=0x7f7ac00fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.931+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3776727441 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac00fe250 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4004940 tx=0x7f7ab4009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:06.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.931+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3776727441 shutdown_connections 2024-01-19T01:23:06.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.931+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3776727441 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac00fe250 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.932 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.931+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3776727441 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ac00fcbf0 0x7f7ac00fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.933 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.931+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3776727441 >> 172.21.15.79:0/3776727441 conn(0x7f7ac00f82d0 msgr2=0x7f7ac00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:06.933 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:06 smithi079 ceph-mon[28374]: pgmap v237: 65 pgs: 65 active+clean; 119 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 9.0 MiB/s wr, 974 op/s 2024-01-19T01:23:06.933 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:06 smithi079 ceph-mon[28374]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:06.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3776727441 shutdown_connections 2024-01-19T01:23:06.934 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3776727441 wait complete. 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 Processor -- start 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 -- start start 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ac00fcbf0 0x7f7ac010d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac010da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac010e080 con 0x7f7ac00fde00 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7ac4cac700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac010e920 con 0x7f7ac00fcbf0 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.932+0000 7f7abdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac010da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7abdd9b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac010da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56362/0 (socket says 172.21.15.79:56362) 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7abdd9b700 1 -- 172.21.15.79:0/3361989839 learned_addr learned my addr 172.21.15.79:0/3361989839 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7abe59c700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ac00fcbf0 0x7f7ac010d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7abdd9b700 1 -- 172.21.15.79:0/3361989839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ac00fcbf0 msgr2=0x7f7ac010d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7abdd9b700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ac00fcbf0 0x7f7ac010d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7abdd9b700 1 -- 172.21.15.79:0/3361989839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4005040 con 0x7f7ac00fde00 2024-01-19T01:23:06.935 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7abdd9b700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac010da80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009380 tx=0x7f7ab4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:06.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7aaeffd700 1 -- 172.21.15.79:0/3361989839 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab400e040 con 0x7f7ac00fde00 2024-01-19T01:23:06.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac010eba0 con 0x7f7ac00fde00 2024-01-19T01:23:06.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7aaeffd700 1 -- 172.21.15.79:0/3361989839 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ab400bd20 con 0x7f7ac00fde00 2024-01-19T01:23:06.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7aaeffd700 1 -- 172.21.15.79:0/3361989839 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab401c630 con 0x7f7ac00fde00 2024-01-19T01:23:06.936 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.933+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac010f0e0 con 0x7f7ac00fde00 2024-01-19T01:23:06.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.935+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac004bb80 con 0x7f7ac00fde00 2024-01-19T01:23:06.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.935+0000 7f7aaeffd700 1 -- 172.21.15.79:0/3361989839 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7ab4013420 con 0x7f7ac00fde00 2024-01-19T01:23:06.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.935+0000 7f7aaeffd700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7aa8068f60 0x7f7aa806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:06.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.935+0000 7f7aaeffd700 1 -- 172.21.15.79:0/3361989839 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7ab40880c0 con 0x7f7ac00fde00 2024-01-19T01:23:06.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.935+0000 7f7abe59c700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7aa8068f60 0x7f7aa806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:06.937 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.936+0000 7f7abe59c700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7aa8068f60 0x7f7aa806b410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00fdc60 tx=0x7f7ab000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:06.940 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:06.940+0000 7f7aaeffd700 1 -- 172.21.15.79:0/3361989839 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7ab4058070 con 0x7f7ac00fde00 2024-01-19T01:23:07.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.190+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7ac0105590 con 0x7f7ac00fde00 2024-01-19T01:23:07.191 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.190+0000 7f7aaeffd700 1 -- 172.21.15.79:0/3361989839 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1809 (secure 0 0 0) 0x7f7ab4057c00 con 0x7f7ac00fde00 2024-01-19T01:23:07.191 INFO:teuthology.orchestra.run.smithi079.stdout:e15 2024-01-19T01:23:07.191 INFO:teuthology.orchestra.run.smithi079.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-01-19T01:23:07.191 INFO:teuthology.orchestra.run.smithi079.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:23:07.191 INFO:teuthology.orchestra.run.smithi079.stdout:legacy client fscid: 1 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:Filesystem 'cephfs' (1) 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:fs_name cephfs 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:epoch 15 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:created 2024-01-19T01:21:01.585027+0000 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:modified 2024-01-19T01:21:18.569403+0000 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:tableserver 0 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:root 0 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:session_timeout 60 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:session_autoclose 300 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:max_file_size 1099511627776 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:required_client_features {} 2024-01-19T01:23:07.192 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure 0 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:last_failure_osd_epoch 46 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:max_mds 1 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:in 0 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:up {0=14472} 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:failed 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:damaged 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:stopped 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:data_pools [3] 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:metadata_pool 2 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:inline_data enabled 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:balancer 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:standby_count_wanted 1 2024-01-19T01:23:07.193 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.qxcrcz{0:14472} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.79:6828/2769248432,v1:172.21.15.79:6829/2769248432] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stdout:Standby daemons: 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi079.cwngbt{-1:14484} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.79:6826/2748689961,v1:172.21.15.79:6827/2748689961] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.hpxjws{-1:24255} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.152:6824/394022976,v1:172.21.15.152:6825/394022976] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stdout:[mds.cephfs.smithi152.btzecy{-1:24265} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.152:6826/2478066701,v1:172.21.15.152:6827/2478066701] compat {c=[1],r=[1],i=[7ff]}] 2024-01-19T01:23:07.194 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7aa8068f60 msgr2=0x7f7aa806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7aa8068f60 0x7f7aa806b410 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00fdc60 tx=0x7f7ab000b040 comp rx=0 tx=0).stop 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 msgr2=0x7f7ac010da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac010da80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4009380 tx=0x7f7ab4009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 shutdown_connections 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f7aa8068f60 0x7f7aa806b410 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ac00fcbf0 0x7f7ac010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 --2- 172.21.15.79:0/3361989839 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f7ac00fde00 0x7f7ac010da80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 >> 172.21.15.79:0/3361989839 conn(0x7f7ac00f82d0 msgr2=0x7f7ac0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 shutdown_connections 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.193+0000 7f7ac4cac700 1 -- 172.21.15.79:0/3361989839 wait complete. 2024-01-19T01:23:07.195 INFO:teuthology.orchestra.run.smithi079.stderr:dumped fsmap epoch 15 2024-01-19T01:23:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.309+0000 7f6b26eab700 1 -- 172.21.15.79:0/2177940946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 msgr2=0x7f6b200fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.309+0000 7f6b26eab700 1 --2- 172.21.15.79:0/2177940946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 0x7f6b200fe250 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c004940 tx=0x7f6b0c009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.309+0000 7f6b26eab700 1 -- 172.21.15.79:0/2177940946 shutdown_connections 2024-01-19T01:23:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.309+0000 7f6b26eab700 1 --2- 172.21.15.79:0/2177940946 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 0x7f6b200fe250 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.309+0000 7f6b26eab700 1 --2- 172.21.15.79:0/2177940946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b200fcbf0 0x7f6b200fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.309+0000 7f6b26eab700 1 -- 172.21.15.79:0/2177940946 >> 172.21.15.79:0/2177940946 conn(0x7f6b200f82d0 msgr2=0x7f6b200fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:07.311 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.310+0000 7f6b26eab700 1 -- 172.21.15.79:0/2177940946 shutdown_connections 2024-01-19T01:23:07.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.310+0000 7f6b26eab700 1 -- 172.21.15.79:0/2177940946 wait complete. 2024-01-19T01:23:07.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.310+0000 7f6b26eab700 1 Processor -- start 2024-01-19T01:23:07.312 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.310+0000 7f6b26eab700 1 -- start start 2024-01-19T01:23:07.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b26eab700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b200fcbf0 0x7f6b2010d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:07.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b26eab700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 0x7f6b2010da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:07.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b26eab700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2010e000 con 0x7f6b200fde00 2024-01-19T01:23:07.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b26eab700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2010e8a0 con 0x7f6b200fcbf0 2024-01-19T01:23:07.313 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b1ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 0x7f6b2010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b24c47700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b200fcbf0 0x7f6b2010d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b1ffff700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 0x7f6b2010da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56382/0 (socket says 172.21.15.79:56382) 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b1ffff700 1 -- 172.21.15.79:0/1263848035 learned_addr learned my addr 172.21.15.79:0/1263848035 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b24c47700 1 -- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 msgr2=0x7f6b2010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b24c47700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 0x7f6b2010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b24c47700 1 -- 172.21.15.79:0/1263848035 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c005040 con 0x7f6b200fcbf0 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.311+0000 7f6b24c47700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b200fcbf0 0x7f6b2010d4c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b200fdc60 tx=0x7f6b1400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.312+0000 7f6b1d7fa700 1 -- 172.21.15.79:0/1263848035 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b14004e90 con 0x7f6b200fcbf0 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.312+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2010eb20 con 0x7f6b200fcbf0 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.312+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2010f060 con 0x7f6b200fcbf0 2024-01-19T01:23:07.314 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.312+0000 7f6b1d7fa700 1 -- 172.21.15.79:0/1263848035 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b1400f040 con 0x7f6b200fcbf0 2024-01-19T01:23:07.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.313+0000 7f6b1d7fa700 1 -- 172.21.15.79:0/1263848035 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b1400c850 con 0x7f6b200fcbf0 2024-01-19T01:23:07.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.313+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b040052e0 con 0x7f6b200fcbf0 2024-01-19T01:23:07.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.314+0000 7f6b1d7fa700 1 -- 172.21.15.79:0/1263848035 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6b14007cc0 con 0x7f6b200fcbf0 2024-01-19T01:23:07.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.314+0000 7f6b1d7fa700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6b10068d60 0x7f6b1006b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:07.315 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.315+0000 7f6b1d7fa700 1 -- 172.21.15.79:0/1263848035 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6b14086940 con 0x7f6b200fcbf0 2024-01-19T01:23:07.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.315+0000 7f6b1ffff700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6b10068d60 0x7f6b1006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:07.316 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.315+0000 7f6b1ffff700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6b10068d60 0x7f6b1006b210 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c013650 tx=0x7f6b0c01c040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:07.318 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.318+0000 7f6b1d7fa700 1 -- 172.21.15.79:0/1263848035 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6b1404da00 con 0x7f6b200fcbf0 2024-01-19T01:23:07.564 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.563+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 --> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b04000bf0 con 0x7f6b10068d60 2024-01-19T01:23:07.565 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.565+0000 7f6b1d7fa700 1 -- 172.21.15.79:0/1263848035 <== mgr.14210 v2:172.21.15.79:6800/655853372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+349 (secure 0 0 0) 0x7f6b04000bf0 con 0x7f6b10068d60 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:5d63039ce1fb346e98993154af5dcfe8a596e67c31136fc03ecd23391ad07bf7", 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout: "in_progress": true, 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout: "which": "Upgrading all daemon types on all hosts", 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout: "services_complete": [], 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout: "progress": "0/21 daemons upgraded", 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout: "message": "Currently upgrading mgr daemons", 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout: "is_paused": false 2024-01-19T01:23:07.566 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6b10068d60 msgr2=0x7f6b1006b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6b10068d60 0x7f6b1006b210 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c013650 tx=0x7f6b0c01c040 comp rx=0 tx=0).stop 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b200fcbf0 msgr2=0x7f6b2010d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b200fcbf0 0x7f6b2010d4c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6b200fdc60 tx=0x7f6b1400b040 comp rx=0 tx=0).stop 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 shutdown_connections 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f6b10068d60 0x7f6b1006b210 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6b200fcbf0 0x7f6b2010d4c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.568 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 --2- 172.21.15.79:0/1263848035 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f6b200fde00 0x7f6b2010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 >> 172.21.15.79:0/1263848035 conn(0x7f6b200f82d0 msgr2=0x7f6b20101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:07.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 shutdown_connections 2024-01-19T01:23:07.569 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.567+0000 7f6b26eab700 1 -- 172.21.15.79:0/1263848035 wait complete. 2024-01-19T01:23:07.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3273603728 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 msgr2=0x7f0d880fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3273603728 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d880fe250 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0d78004940 tx=0x7f0d78009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:07.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3273603728 shutdown_connections 2024-01-19T01:23:07.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3273603728 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d880fe250 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.694 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3273603728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d880fcbf0 0x7f0d880fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3273603728 >> 172.21.15.79:0/3273603728 conn(0x7f0d880f82d0 msgr2=0x7f0d880fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3273603728 shutdown_connections 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.692+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3273603728 wait complete. 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8f8e0700 1 Processor -- start 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8f8e0700 1 -- start start 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8f8e0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d880fcbf0 0x7f0d88071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8f8e0700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d8806ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8f8e0700 1 -- --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88070590 con 0x7f0d880fde00 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8f8e0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88070700 con 0x7f0d880fcbf0 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8ce7b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d8806ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8ce7b700 1 --2- >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d8806ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.79:3300/0 says I am v2:172.21.15.79:56406/0 (socket says 172.21.15.79:56406) 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8ce7b700 1 -- 172.21.15.79:0/3344125262 learned_addr learned my addr 172.21.15.79:0/3344125262 (peer_addr_for_me v2:172.21.15.79:0/0) 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.693+0000 7f0d8d67c700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d880fcbf0 0x7f0d88071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d8ce7b700 1 -- 172.21.15.79:0/3344125262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d880fcbf0 msgr2=0x7f0d88071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:07.695 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d8ce7b700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d880fcbf0 0x7f0d88071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:07.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d8ce7b700 1 -- 172.21.15.79:0/3344125262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78005040 con 0x7f0d880fde00 2024-01-19T01:23:07.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d8ce7b700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d8806ff90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0d78009380 tx=0x7f0d78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:07.696 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d7dffb700 1 -- 172.21.15.79:0/3344125262 <== mon.0 v2:172.21.15.79:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7800e040 con 0x7f0d880fde00 2024-01-19T01:23:07.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d88070980 con 0x7f0d880fde00 2024-01-19T01:23:07.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d7dffb700 1 -- 172.21.15.79:0/3344125262 <== mon.0 v2:172.21.15.79:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d7800bd20 con 0x7f0d880fde00 2024-01-19T01:23:07.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d7dffb700 1 -- 172.21.15.79:0/3344125262 <== mon.0 v2:172.21.15.79:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7801c630 con 0x7f0d880fde00 2024-01-19T01:23:07.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.694+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d881155b0 con 0x7f0d880fde00 2024-01-19T01:23:07.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.695+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8804bb80 con 0x7f0d880fde00 2024-01-19T01:23:07.697 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.696+0000 7f0d7dffb700 1 -- 172.21.15.79:0/3344125262 <== mon.0 v2:172.21.15.79:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0d78013420 con 0x7f0d880fde00 2024-01-19T01:23:07.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.697+0000 7f0d7dffb700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0d74068f60 0x7f0d7406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-19T01:23:07.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.697+0000 7f0d7dffb700 1 -- 172.21.15.79:0/3344125262 <== mon.0 v2:172.21.15.79:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0d780880b0 con 0x7f0d880fde00 2024-01-19T01:23:07.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.697+0000 7f0d8d67c700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0d74068f60 0x7f0d7406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-19T01:23:07.698 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.698+0000 7f0d8d67c700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0d74068f60 0x7f0d7406b410 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d880718a0 tx=0x7f0d84009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-19T01:23:07.703 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:07.703+0000 7f0d7dffb700 1 -- 172.21.15.79:0/3344125262 <== mon.0 v2:172.21.15.79:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0d78058060 con 0x7f0d880fde00 2024-01-19T01:23:07.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:07 smithi152 ceph-mon[35694]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:07.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:07 smithi152 ceph-mon[35694]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:07.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:07 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/2276391616' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T01:23:07.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:07 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3361989839' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:23:07.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:07 smithi079 ceph-mon[28374]: from='client.14614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:07.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:07 smithi079 ceph-mon[28374]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:07.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:07 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/2276391616' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-01-19T01:23:07.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:07 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3361989839' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-01-19T01:23:08.117 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.117+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 --> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0d88105590 con 0x7f0d880fde00 2024-01-19T01:23:08.118 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.117+0000 7f0d7dffb700 1 -- 172.21.15.79:0/3344125262 <== mon.0 v2:172.21.15.79:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+201 (secure 0 0 0) 0x7f0d78057bf0 con 0x7f0d880fde00 2024-01-19T01:23:08.118 INFO:teuthology.orchestra.run.smithi079.stdout:HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-01-19T01:23:08.118 INFO:teuthology.orchestra.run.smithi079.stdout:[WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-01-19T01:23:08.118 INFO:teuthology.orchestra.run.smithi079.stdout: fs cephfs has deprecated feature inline_data enabled. 2024-01-19T01:23:08.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0d74068f60 msgr2=0x7f0d7406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:08.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0d74068f60 0x7f0d7406b410 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d880718a0 tx=0x7f0d84009040 comp rx=0 tx=0).stop 2024-01-19T01:23:08.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 msgr2=0x7f0d8806ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-19T01:23:08.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d8806ff90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0d78009380 tx=0x7f0d78009e30 comp rx=0 tx=0).stop 2024-01-19T01:23:08.120 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 shutdown_connections 2024-01-19T01:23:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:6800/655853372,v1:172.21.15.79:6801/655853372] conn(0x7f0d74068f60 0x7f0d7406b410 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d880fcbf0 0x7f0d88071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 --2- 172.21.15.79:0/3344125262 >> [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] conn(0x7f0d880fde00 0x7f0d8806ff90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-19T01:23:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.119+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 >> 172.21.15.79:0/3344125262 conn(0x7f0d880f82d0 msgr2=0x7f0d88101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-19T01:23:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.120+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 shutdown_connections 2024-01-19T01:23:08.121 INFO:teuthology.orchestra.run.smithi079.stderr:2024-01-19T01:23:08.120+0000 7f0d8f8e0700 1 -- 172.21.15.79:0/3344125262 wait complete. 2024-01-19T01:23:08.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:08 smithi152 ceph-mon[35694]: pgmap v238: 65 pgs: 65 active+clean; 119 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.1 MiB/s wr, 581 op/s 2024-01-19T01:23:08.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:08 smithi152 ceph-mon[35694]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:08.924 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:08 smithi152 ceph-mon[35694]: from='client.? 172.21.15.79:0/3344125262' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-01-19T01:23:08.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:08 smithi079 ceph-mon[28374]: pgmap v238: 65 pgs: 65 active+clean; 119 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.1 MiB/s wr, 581 op/s 2024-01-19T01:23:08.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:08 smithi079 ceph-mon[28374]: from='client.24383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-01-19T01:23:08.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:08 smithi079 ceph-mon[28374]: from='client.? 172.21.15.79:0/3344125262' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-01-19T01:23:10.923 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:10 smithi152 ceph-mon[35694]: pgmap v239: 65 pgs: 65 active+clean; 130 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6.1 MiB/s wr, 717 op/s 2024-01-19T01:23:10.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:10 smithi079 ceph-mon[28374]: pgmap v239: 65 pgs: 65 active+clean; 130 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 6.1 MiB/s wr, 717 op/s 2024-01-19T01:23:12.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:12 smithi152 ceph-mon[35694]: pgmap v240: 65 pgs: 65 active+clean; 141 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.0 MiB/s wr, 736 op/s 2024-01-19T01:23:12.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:12.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:12.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:12 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:23:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:12 smithi079 ceph-mon[28374]: pgmap v240: 65 pgs: 65 active+clean; 141 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.0 MiB/s wr, 736 op/s 2024-01-19T01:23:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:12.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:12 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-19T01:23:14.928 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:14 smithi152 ceph-mon[35694]: pgmap v241: 65 pgs: 65 active+clean; 145 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.3 MiB/s wr, 740 op/s 2024-01-19T01:23:14.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:14 smithi079 ceph-mon[28374]: pgmap v241: 65 pgs: 65 active+clean; 145 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.3 MiB/s wr, 740 op/s 2024-01-19T01:23:16.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:16 smithi152 ceph-mon[35694]: pgmap v242: 65 pgs: 65 active+clean; 160 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.3 MiB/s wr, 749 op/s 2024-01-19T01:23:16.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:16.673 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:16 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:16.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:16 smithi079 ceph-mon[28374]: pgmap v242: 65 pgs: 65 active+clean; 160 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.3 MiB/s wr, 749 op/s 2024-01-19T01:23:16.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:16.998 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:16 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:17.411 INFO:tasks.workunit.client.1.smithi152.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-01-19T01:23:17.423 INFO:tasks.workunit.client.1.smithi152.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-01-19T01:23:17.423 INFO:tasks.workunit.client.1.smithi152.stderr:+ make 2024-01-19T01:23:18.069 INFO:tasks.workunit.client.1.smithi152.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-01-19T01:23:18.153 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:18.153 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:Jan 19 01:23:17 smithi152 ceph-mon[35694]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:18.208 INFO:tasks.workunit.client.0.smithi079.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-01-19T01:23:18.208 INFO:tasks.workunit.client.0.smithi079.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-01-19T01:23:18.209 INFO:tasks.workunit.client.0.smithi079.stderr:+ make 2024-01-19T01:23:18.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:18.248 INFO:journalctl@ceph.mon.smithi079.smithi079.stdout:Jan 19 01:23:17 smithi079 ceph-mon[28374]: from='mgr.14210 172.21.15.79:0/390552004' entity='mgr.smithi079.miqczk' 2024-01-19T01:23:18.375 INFO:tasks.workunit.client.0.smithi079.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-01-19T13:04:17.824 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-01-19T13:04:19.985 DEBUG:teuthology.task.console_log:Killing console logger for smithi079 2024-01-19T13:04:20.818 DEBUG:teuthology.task.console_log:Killing console logger for smithi152 2024-01-19T13:04:20.822 DEBUG:teuthology.exit:Finished running handlers